Identifying Credible Sources of Health Information in Social Media: Principles and Attributes

July 16, 2021

Disclaimer: The views expressed in this paper are those of the authors and not necessarily of the authors’ organizations, the National Academy of Medicine (NAM), the National Academies of Sciences, Engineering, and Medicine (the National Academies), or the National Institutes of Health (NIH). The paper is intended to help inform and stimulate discussion. It is not a report of the NAM or the National Academies.

ABSTRACT | Social media is widely used as a source of health information for the general public. The potential for information shared through social media to infl uence health outcomes necessitates action by social media platforms to enhance access and exposure to high-quality, science-based information. This paper summarizes the work of an independent advisory group convened by the National Academy of Medicine that deliberated and gathered information to develop a set of initial principles and attributes that could inform platforms’ identifi cation and possible elevation of credible sources of health information. Using these principles and attributes as a framework, the authors discuss the likelihood of credibility among major categories and types of nonprofi t and government organizations that share health information through social media. The authors also emphasize the need for parallel strategies in addition to source evaluation, including assessment of content, as well as important ethical considerations such as the protection of free speech and individual autonomy. The paper also stresses that, in order to be considered credible themselves, social media platforms should share data with behavioral and public health researchers to understand the eff ects of such policies on both online and offl ine behaviors.

Introduction

People seek, share, and receive health information from a wide variety of sources, such as health care professionals, insurance and pharmaceutical companies, family and friends, media, educational materials, advertisements, and the internet—including social media. Increasing numbers of Americans have turned to internet sources for health and medical information in recent years, with approximately three out of four searching for health information online today, and similar rates among Europeans [1,2]. However, both high- and low-quality health information can be found online, and few social media platforms (SMPs) [a] differentiate between credible and non-credible sources of information. Consequentially, consumers must make their own judgments about how much trust to place in a source and the quality of the information it shares. These judgments are influenced by their level of health and digital literacy, prior knowledge, personal situations, and personal beliefs [3].

“Misinformation” is that which conflicts with the best scientific evidence available at the time. “Disinformation” describes a “coordinated or deliberate” effort to spread misinformation in order to gain “money, power, or reputation” [1]. Social media allows both misinformation and disinformation to be disseminated much more rapidly and broadly than ever before [4]. The ability for people to tailor their preferences on SMPs to see information from only the sources they select raises concerns about “bubbles” or “echo chambers” that could reinforce existing beliefs (although recent research has challenged this notion [5]). However, consumers do not have to proactively seek information that confirms their beliefs; algorithms used by SMPs and other web platforms often recommend content on the basis of users’ past behaviors and expressed interests, leading to passive or incidental exposure [6]. In the case of low-quality health information, such reinforcement loops can be harmful.

The coronavirus disease 2019 (COVID-19) pandemic has demonstrated the potentially malign outcomes of this aspect of social media. Misinformation about the disease spread through social media and other online forums—often fueled by politicization of scientific information—has considerably harmed the adoption of recommended prevention and control behaviors and has decreased support for vital policies, such as vaccination [7]. Therefore, SMPs are capable of amplifying misinformation and disinformation in harmful ways, including those that may lead to poor outcomes for individual as well as population health [8]. The authors believe that these platforms have an important opportunity—and a growing responsibility—to intervene, not only to counteract these harmful trends but also to enhance consumers’ access and exposure to high-quality, science-based health information. Proactive interventions by SMPs are one potential approach, although not a sole solution, to the challenge of “platform governance,” an issue that has been the subject of increasing policy debate [9].

The tremendous reach of SMPs among broad and diverse audiences affords them unique potential to support health-promoting behaviors amid the COVID-19 pandemic, as well as other current and future health challenges. For example, the two current most popular SMPs used by organizations to share health information—Facebook and YouTube—reach 2.85 billion [10] and “over 2 billion,” [11] monthly active users, respectively [b]. This represents a significant portion of the world’s population, estimated by the U.S. Census Bureau to be nearly 7.8 billion people in June 2021 [12]. Harnessing the power of social media to elevate high-quality information could therefore have a truly trans-formative effect on health and well-being worldwide.

However, determining what constitutes high-quality health information is a complex and multidimensional process. Although SMPs are beginning to pilot strategies to elevate and label high-quality information, there are no public data available to demonstrate what works and no scientific or technical consensus about the most effective approach. Nevertheless, the urgency of ensuring access to high-quality health information necessitates action, even if such action is initially imperfect. The challenge will require collaboration among public and private actors to develop incremental and iterative solutions, with attention to transparency, accountability, and incorporation of feedback from a diverse set of stakeholders.

This paper focuses specifically on the evaluation of sources of health information, rather than content or design (discussed further under “Scope”). The authors offer initial principles and attributes for consideration by SMPs in their efforts to identify credible sources—with the ultimate goal of promoting access to high-quality health information. The guidance in this paper is therefore limited in scope and is offered as a starting point in what should be an ongoing process. This guidance will also need to be regularly revisited and updated according to changes in the online information ecosystem. SMPs should invest in ongoing, rigorous research and analysis of this subject; commit to transparency and continuous quality improvement; and build and sustain collaborations with scientific, health, ethical, and other communities to ensure an effective and accountable approach.

Although this paper is intended to inform the policies of SMPs, organizations that share health information through social media may find the principles and credibility attributes useful in assessing their own approach. Importantly, members of the public might also use this guidance to inform their personal evaluation of sources. Both groups should be engaged by SMPs and others seeking to improve the accessibility of high-quality health information in social media.

Background

In March 2021, the National Academy of Medicine (NAM) launched a project to help identify principles for identifying credible sources of health information in social media, of which this paper is the principal output. Sponsored by YouTube’s Healthcare and Public Health Partnerships arm [c], the project was inspired by the goal of enhancing public access to evidence-based health information during the COVID-19 pandemic, although the issue has relevance beyond the current crisis.

The project involved an independent expert advisory group composed of multi-disciplinary experts in information governance, health information development, public health and health equity, social media and misinformation, and science communication (members of which also authored this paper), a public webinar, a public comment period, and other information-gathering activities. This paper does not constitute official recommendations from the NAM or the National Academies of Sciences, Engineering, and Medicine (NASEM), nor does it represent an endorsement of any actions taken by YouTube or other SMPs following its publication.

Methods

Managing Conflict of Interest

The NAM is an organization whose influence stems in part from its reputation as a credible source of health information. Further, the NAM disseminates this information in part through social media [d]. To minimize conflict of interest (COI), the NAM took steps to ensure the independence and objectivity of the advisory group and this paper. This paper represents the opinions of the authors and does not reflect a consensus position of the NAM, NASEM, or the authors’ organizations. The authors did not receive payment from the NAM, NASEM, or YouTube for their contributions to this paper, and the authors’ declared individual COIs are included in this paper’s back matter. This paper has been revised in response to scientific peer review by individuals who were chosen for their expertise in social media, ethics, health literacy, law, communications, and policy but are unknown to the authors.

Deliberative Sessions

The authors met for four closed, deliberative sessions between March and June 2021. Representatives from YouTube attended the first 60 minutes of the initial session in order to explain the company’s current policies and future goals with regard to elevating high-quality health information and to answer questions from the authors. Representatives from YouTube did not attend any part of the subsequent deliberative sessions. Notes from all three sessions are available to the public on the project webpage: NAM.edu/AuthoritativeHe-althSources.

Information-Gathering Public Webinar

On April 5, 2021, the NAM hosted a public webinar to gather information to inform the authors’ deliberations. The webinar was planned with the input of the authors, and all authors attended. The topics covered included background on YouTube’s goals with regard to elevating credible sources of health information; the health and social consequences of social media misinformation and disinformation; how health information is received at the community level; and unintended consequences of social media content moderation strategies. The session concluded with a question-and-answer session among the authors and presenters (see Box 1). The webinar was attended by approximately 400 members of the public. The webinar recording, transcript, slide presentations, and a written summary are available on the project webpage, along with a synthesis of questions and comments submitted by public attendees.

Box 1. Takeaway Points from the NAM Webinar on “Defining the Authority of Online Providers of Health Information”.

• Scientific and medical collaborations with social media companies offer a unique opportunity to share high-quality health information with a broad audience.

• People are resourceful when seeking health information, meaning that they consult many sources and often will not settle for the word of a single, “authoritative” entity.

• High-quality health information must be understandable, engaging, and culturally competent [a] to be influential. Health and medical organizations must work hard to gain the trust of people at the community level.

• Inter-science debate and evolving information makes social media content moderation very challenging. For example, during the COVID-19 pandemic, information shared by “authoritative” sources such as the World Health Organization and the Centers for Disease Control and Prevention quickly became outdated.

• A perfect solution to this challenge may not exist, but achieving something “better than” the status quo is a worthy goal.

Preliminary Discussion Document and Public Comment Period

The authors created a four-page preliminary discussion document to solicit feedback on the project from interested parties, including researchers, providers of online health information, and members of the public. The document contained background on the project; preliminary definitions and source categories; and ethical, logistical, and public health considerations. The document was posted on the project webpage on April 5, 2021, where it remains available [13].

The NAM hosted a questionnaire to collect comments on the discussion document between 12:00 pm ET on April 5, 2021, and 11:59 pm ET on April 9, 2021 (see Appendix C). The comment opportunity was promoted via email to approximately 1,000 individuals who had registered to attend the webinar and/or signed up for the project mailing list, as well as shared through the NAM’s social media channels. In total, the NAM received 49 comments. Fourteen of the commenters provided feedback on behalf of an organization, while the remainder commented as individuals. Three commenters were from Canada, one was from Mexico, one was from Egypt, and the remainder were from the United States. The comments were analyzed, sorted into themes, and summarized by a contractor [e]; this synthesis is available on the project webpage and presented more briefly in Box 2. The authors reviewed all comments received and considered them in developing this paper.

Box 2. Key Themes Among Feedback Received During the Public Comment Period.

• The need for parallel strategies to supplement elevation of credible health sources, including content analysis; reduction of misinformation; consumer education/efforts to increase health and digital literacy; and efforts to improve the communication skills of credible sources, including a focus on cultural competence [a].

• The need to consider the evolving nature of science and health and medical practice; the necessity of experimentation and iteration should not impact perception of a source’s credibility. On the other hand, steps must be taken to increase consumers’ understanding of the limitations of information, particularly primary or preliminary research.

• The importance of conflict of interest disclosures; even the most credible sources have conflicts, and no organization should be exempt from disclosing them.

• The need to protect free speech and personal autonomy; efforts to elevate credible health sources should not amount to censorship of any other sources, and individual consumers should retain the ability to make their own judgments.

• The desire for a simple system by which users can evaluate source credibility, such as a “visual seal of approval” or a “1-5 rating system.”

Review of Existing Models for Evaluation of Source Credibility

The authors performed a scan of existing models for evaluating source credibility and/or information quality (see Box 3 and Appendix A). Major themes that emerge across these models include the importance of independence from profit motivations and bias; rigorous content review processes; transparency and accountability; and mission-driven policies.

Box 3. Models for Evaluation of Source Credibility.

Clinical Practice Guidelines We Can Trust. This 2011 Institute of Medicine consensus report made recommendations for identifying high-quality clinical practice guidelines (CPGs) among the nearly 27,000 then contained in the National Guideline Clearinghouse. The report committee concluded that certifying organizations with trustworthy CPG development procedures, rather than evaluating the content each individual CPG, was a reasonable (although not the only) approach to the challenge.

The CRAP Test. Developed by librarian Molly Beestrum, the CRAP Test is a system for evaluating the credibility of a website according to four major attributes: Currency/Credibility, Reliability, Authority, and Purpose/Point of View. Embedded within each of these attributes are questions such as, How recent is the information? Does the website include citations? What are the author’s credentials? Does the author seem to be trying to push an agenda or sell you something? Educator Mike Caulfield has developed an alternative to the CRAP model called SIFT (Stop. Investigate the Source. Find Better Coverage. Trace claims, quotes, and media to the original context), which is designed to help “students get better at sorting truth from fiction from everything in between” [a].

Health on the Net (HON) Foundation Certification. HON is an international nonprofit organization based in Switzerland. HON certification holds health and medical websites accountable to basic ethical standards in the presentation of information, including sharing information from only trained and qualified professionals, respecting patient and consumer privacy, providing evidence in support claims, and disclosing financial interests, among others. Websites with HON certification earn the right to display a visual seal as an indication of their integrity.

MEDLINE and MedlinePlus (National Library of Medicine [NLM]). MEDLINE is an NLM database with over 27 million references to journal articles in the life sciences. To decide which journals (i.e., article sources) to include, MEDLINE applies a set of criteria including scope and coverage, editorial policies and processes, scientific and methodological rigor, production and administration, and impact. MEDLINE selection also depends on the judgment of an independent Literature Selection Technical Review Committee, a Federal Advisory Committee.

MedlinePlus is an NLM website designed to share health information with the public. MedlinePlus primarily links to other government websites but will consider inclusion of nongovernment websites (i.e., information sources) if they demonstrate a mission to share high-quality health information; display transparency and trustworthiness; provide unbiased content for the purpose of education; and ensure the accessibility of information, among other criteria. MedlinePlus also gives preference to websites that do not host advertisements.

URAC Certification for Health Content Providers and Health Websites. URAC is an accreditor that offers certifications for health information sources that meet standards for disclosures, editorial and content review processes, privacy and security, external linking policies, consumer complaint processes, and more.

Scope

Given the complexity of the task—including the volume of health information shared through social media and the controversial nature of evolving content moderation policies—the authors limit their guidance to what they believe is a feasible first step toward enhancing access to high-quality health information. Therefore, this paper focuses on the credibility of sources of health information, rather than the information shared by these sources. Source evaluation is a common means of curating a large volume of content because it eliminates the need to evaluate every piece of information individually (although advances in machine learning may soon increase the feasibility of large-scale content evaluation).

However, although a reasonable place to begin, source evaluation cannot yield a complete solution to the challenge of increasing access to high-quality health information in social media. The credibility of a source is, at most, an indicator of information quality and by no means a guarantee. Furthermore, even organizations with strong reputations for credibility are not infallible. For example, the World Health Organization (WHO) and the Centers for Disease Control and Prevention (CDC) were slow to acknowledge the role of airborne transmission in the COVID-19 pandemic, and the CDC recently removed three pieces of guidance related to the virus from its website for failing to adhere to the agency’s scientific standards [14, 15]. Therefore, the authors stress that identifying credible sources of information is a starting point only and must be supplemented by ongoing and iterative efforts to assess the quality of information.

The authors further limit their consideration in this paper to government and nonprofit organizations (including nonprofit news sources that share health information), not individuals (e.g., independent physicians with Facebook pages) or for-profit companies. Individuals require separate analysis because they lack the organizational infrastructure that is the basis of the authors’ approach to source evaluation in this paper. For-profits have a unique set of financial interests that, likewise, require a separate assessment.

However, the authors’ decision to omit consideration of individual and for-profit sources from this paper does not reflect a judgment of their credibility. Individuals and for-profits may be highly credible and are influential sources of health information in social media. Therefore, an assessment of their credibility, as well as the quality of the information they share, should be the focus of future analysis. It also should be noted that demarcations between individuals and organizations and between nonprofit and for-profit organizations are not always clear. Many of the principles laid out in this paper can apply to for-profits.

Finally, the authors have limited their present focus to organizations based in the United States, including those that provide information in languages other than English. While some of the guidance in this paper can be applicable to organizations outside the United States, additional research and the engagement of global partners will be needed for this work in the international context.

Key Terms

Terminological precision is critical to this task and was the subject of careful deliberation by the authors. The following are definitions and discussions of the key terms used in this paper.

Credible

For the purposes of this paper, the authors present their own definition of credible in the context of sources of online health information: “offering information that is consistent with the best scientific evidence available at the time and employing processes to reduce conflict of interest and promote transparency and accountability.” The principles that inform this definition are explained in the following section.

High-Quality Information

As noted in the Introduction, high-quality information is that which is “science-based” or consistent with the best scientific evidence available at the time. The state of science and knowledge is always evolving, so the marker of time is an important component of this definition. The evolution of knowledge is also the reason that more absolute terms, such as accurate, are less appropriate. Although this paper does not consider information quality directly, increasing access to high-quality information is the goal of the approach under discussion.

Health Information

The authors define health information as content pertaining to health conditions (physical and mental), behaviors affecting health, public health, population health, health care, health policy, or biomedical science.

Source

For the purposes of this paper, a source is an entity that offers health information through one or more social media channels branded to that entity. A channel is a proprietary forum where a source can share content (text, visual, video, or audio) and interact with social media users who choose to “follow” or “subscribe” to that channel, as well as users who discover the content through search engines or SMPs’ “recommended content” algorithms.

Credible Source of Health Information

Building on the definitions and discussion previously mentioned, the authors define credible source of health information as “a source that is likely to offer high-quality information and employ processes to reduce conflict of interest and promote transparency and accountability.” The use of the word “likely” in this definition reinforces the notion that source credibility does not necessarily equate to information quality, yet is still a useful indicator for consumers.

Foundational Principles

On the basis of their information gathering and deliberation, the authors developed the following foundational principles to guide identification of credible sources of health information in social media.

Principle 1: Science-Based

Sources should provide information that is consistent with the best scientific evidence available at the time and meet standards for the creation, review, and presentation of scientific content.

This principle reflects the authors’ conviction that scientific evidence is the only reliable predictor of health outcomes and therefore should be the foundation of health information provided to consumers. There are a number of attributes (e.g., use of citations) that help to indicate whether a source is sharing information that is consistent with the best scientific evidence available at the time, described in the following section.

Principle 2: Objective

Sources should take steps to reduce the influence of financial and other forms of conflict of interest or bias that might compromise or be perceived to compromise the quality of the information they provide.

This principle acknowledges that all sources have COIs or inherent biases. However, in order to be considered credible, sources should strive to separate the presentation of health information from profit motives and other biases (e.g., political). Sources should also disclose conflicts, as noted in the next principle.

Principle 3: Transparent and Accountable

Sources should disclose the limitations of the information they provide, as well as conflicts of interest, content errors, or procedural missteps.

The final principle acknowledges the fallibility of both organizations—which cannot eliminate COI and errors—and science itself. At the frontiers of understanding, scientific knowledge changes over time as more evidence becomes available and as existing evidence is analyzed in new ways. Scientific evidence, no matter how rigorous, can never guarantee a certain outcome for every individual or every context. Furthermore, Black, Indigenous, and People of Color (BIPOC) and other groups, such as LGBTQIA+ individuals and people with disabilities, are underrepresented within organizations traditionally considered authorities in science, meaning that the best available science might not fully reflect their experiences (discussed further in “Structural Bias”).

To maintain credibility, sources must clearly acknowledge the limitations of the information they share so that consumers can reach fully informed conclusions. Fundamentally, this last principle reflects one of the key themes among the public comments the authors received—the importance of protecting the right of individuals to autonomy and independent evaluation of the information they consume and the sources they choose to trust. It also acknowledges sources’ right to freedom of speech [f], but at the same time, requires sources to be fully transparent and provide all the context necessary for consumers to reach an informed judgment. However, protection of free speech and consumer autonomy must be balanced against the harms of misinformation and disinformation, as well as recent anti-science and “post-truth” trends in the media [16]. “Post-truth” refers to an environment in which scientific evidence is disregarded by some in favor of an alternative set of beliefs [17].

Credibility Attributes

Using the foundational principles as a scaffold, the authors identified a set of attributes that generally describe credible sources of health information (see Table 1). Not every source can display every attribute, but this should not preclude a general assessment of credibility. For example, a professional association may have a lobbying arm, which is counter to one of the attributes under the “objective” principle. However, the same organization might have a research arm that nearly or fully aligns with the attributes under the “science-based” principle. Furthermore, this organization may clearly disclose its lobbying activities to the public and maintain a strict firewall between political messages and health information for the public, thereby aligning with attributes under the “transparent and accountable” principle.

TABLE 1. Foundational Principles and Attributes of Credible Sources of Health Information.

Foundational Principle	Attributes
Science-Based: Sources should provide information that is consistent with the best scientific evidence available at the time and meet standards for the creation, review, and presentation of scientific content.	Acknowledges the limitations and evolution of knowledge (e.g., early or incomplete knowledge, as seen in the COVID-19 pandemic; small sample size; correlation versus causation, etc.) Clearly labels information with the date it was last updated and strives to reassess and update content Demonstrates subject-specific expertise (i.e., consistent and well-regarded contributions in a given field) Links to and is linked to by other credible sources [a] Provides citations for information shared and evidence to justify claims Synthesizes information from multiple sources, rather than a single source Uses a consensus process to develop the information shared [b] Uses peer review or another form of content review to vet information before sharing [c]
Objective: Sources should take steps to reduce the influence of financial and other forms of conflict of interest (COI) or bias that might compromise or be perceived to compromise the quality of the information they provide.	Keeps health information separate from financial, political, or ideological messages Maintains independence from funders [d] Separates lobbying activities from health information (or does not engage in lobbying) Does not include advertisements with relevant health information (or does not host advertisements at all) [e]
Transparent and Accountable: Sources should disclose the limitations of the information they provide, as well as conflicts of interest, content errors, or procedural missteps.	Discloses financial and nonfinancial conflicts Discloses relevant policy positions and lobbying activities Follows FACA regulations or similar transparency policies [f] Posts public corrections or retractions Prioritizes accessibility and equitable access to information Provides a mechanism for public feedback Shares data, methods, or draft recommendations

^[a]For example, an organization could seek public comments on an interim set of health guidelines before finalizing and sharing the information more broadly.

^[b]A consensus process involves assembling a group of experts with diverse perspectives who assess a body of evidence and deliberate in order to arrive at an opinion or guidance that reflects the consensus of the group.

^[c]A peer review process involves sharing the draft of a publication or other product with reviewers who have expertise or experience in the given topic and can provide feedback as to the product’s accuracy, balance, and appropriateness.

^[d]For example, an academic journal could maintain editorial independence (i.e. sole authority over published content) from the organization that funds it.

^[e]For example, an organization might host an advertisement for a cancer drug but keep this advertisement separate from the information it shares about cancer.

^[f]FACA stands for the Federal Advisory Committee Act, which established requirements for committees that advise the federal government. These requirements include public access to meetings and meeting notes, as well as summaries of expenditures (https://www.gsa.gov/policy-regulations/policy/federal-advisory-committee-management/advice-and-guidance/the-federal-advisory-committee-act-faca-brochure)

A credible source should demonstrate a preponderance of the attributes listed in Table 1 but should not be required to meet a formal numerical threshold. Although one of the key themes among the public comments that informed this paper was the desire for a simple rating system, the authors believe that such a calculus would be inappropriate given that each attribute is not necessarily of equal weight or importance. Instead, SMPs and consumers of health information could consider these principles and attributes as a framework to inform their own assessments of a source’s credibility. Further, sources of health information could consider using Table 1 as a roadmap to assess and potentially enhance their own credibility.

To avoid perfection paralysis, the authors believe that general alignment with the principles and attributes listed in Table 1, coupled with full disclosure of any deviations, could serve as a reliable initial signal of a source’s credibility. As noted in the sections that follow, some types of sources are subject to pre-existing, standardized vetting mechanisms that signal such alignment. However, there remain credibility concerns with these source types as a whole. All sources should publicly disclose deviations from the principles and attributes and be subject to other strategies to ensure information quality (described later in this paper).

Identifying Credible Sources of Health Information

Categorization

A very wide range of U.S. nonprofit or government sources provide health information through social media, including professional societies; health care organizations; public health departments; universities; think tanks; philanthropies; medical journals; grassroots community organizations; state, local, tribal, and territorial government health agencies, and more. The scope and size of these sources varies dramatically. Evaluating the credibility of each one individually, while a worthy eventual goal, is infeasible as a first step. Therefore, just as the evaluation of source credibility (versus information quality) is offered as an entry point into a larger task, the assessment of source categories (versus individual sources) is an initial tactic to assess credibility at scale.

Leveraging Pre-Existing, Standardized Vetting Mechanisms

Several categories of sources are subject to pre-existing, standardized vetting mechanisms that indicate general alignment with the authors’ foundational principles and attributes. Therefore, SMPs could consider sources in these categories as likely to be credible. However, these vetting mechanisms are not an exact match for the principles and attributes and reflect varying degrees of rigor. In addition, there are credibility concerns associated with each of these categories, as noted in the sections that follow. Therefore, sources in these categories should undergo additional vetting by SMPs (and also, perhaps, by independent third parties, as is discussed later), including some form of content review, as they refine their approaches to source evaluation.

Accredited Organizations

Accreditation is a voluntary process by which an organization earns a formal qualification as proof of its ability to meet quality and performance standards set by an accreditor. Although processes vary considerably by accreditor, accreditation is generally an intensive evaluation that involves extensive documentation and a site visit. Accreditation standards are set on the basis of research and evidence that demonstrates which qualities are associated with the highest degree of organizational effectiveness and the best possible outcomes for patients and other beneficiaries.

Because organizations must apply for reaccreditation on a regular basis (e.g., every few years), maintenance of accreditation indicates an ongoing commitment to transparency and accountability. Accreditors provide publicly accessible lists of organizations that have earned accreditation, as well as, in some cases, organizations that have failed to earn or maintain accreditation.

Accreditors themselves engage in continuous quality improvement and are held to a high standard of performance based on their recognition by federal authorities such as the Department of Education. In some cases, accreditors may be sponsored by relevant professional associations, which can suggest policy changes that ensure up-to-date accreditation criteria. For example, the Association of American Medical Colleges and the American Medical Association jointly sponsor the Liaison Committee on Medical Education, which accredits medical schools.

In another example, the Centers for Medicare & Medicaid Services (CMS) awards “deeming authority” to approved health care organization accreditors. Deeming authority means that accreditation can substitute for inspection by a state agency to determine whether a health care organization is eligible to receive payment from Medicare and Medicaid programs [18]. Similarly, the U.S. Department of Education provides a list of accreditors that are “recognized by the Secretary as reliable authorities concerning the quality of education or training offered by the institutions of higher education or higher education programs they accredit” [19].

Together, accredited organizations, accreditors, and collaborator organizations function as a network that supports consistent and high performance standards, continuous evaluation and improvement, and public transparency and accountability—although these characteristics are not specific to the context of sharing health information through social media.

Categories of accredited organizations that serve as sources of health information for the public include educational institutions (universities and health professions schools), health care organizations, health plans, and public health departments (see Box 5 for a summary and Appendix Table B-1 for a list of accreditors and what accreditation signifies for organizations in each category).

Box 5. Types of Nonprofit, Accredited Organizations That Serve as Sources of Health Information in the United States.

Health Professions Schools and Other Educational Institutions

• Schools of medicine

• Schools of nursing

• Schools of public health

• Schools of dentistry

• Schools of pharmacy

• Universities (e.g., departments or schools of biomedical science, health policy, social science, etc.)

Nonprofit Health Plans

Public Health Departments

• State

• Tribal

• Territorial

• Local

• U.S. military installation

Health Care Organizations

• Ambulatory care providers (e.g., behavioral health centers, birthing centers, dental offices, dialysis centers, medical offices, occupational health centers, outpatient surgical centers, pain management centers, podiatry centers, radiation oncology practices, student health clinics, urgent care practices, women’s health clinics, etc.)

• Home-based health care providers

• Hospitals (e.g., academic medical centers, acute care and long-term care hospitals, critical access hospitals, and rehabilitation facilities. There are also a wide variety of specialty hospitals, including cancer, stroke, and cardiac centers; children’s hospitals; psychiatric hospitals; women’s hospitals, etc.)

NOTES: For-profit health plans are not considered in this paper. Public health departments are also government organizations, which are subject to additional transparency and accountability rules. Public health department accreditation is a relatively new field, and accreditation is issued by a single body, the Public Health Accreditation Board.

Credibility Concerns

As noted above, the accreditation mechanisms discussed in this section do not evaluate an organization’s credibility as a source of health information in social media. Rather, they affirm the general credibility of an organization in its role as a provider of a specific service, such as education or health care. Accredited organizations may not always adhere to the authors’ credibility principles and criteria (nonprofit health plans and ambulatory care centers, for example, may be subject to strong profit motives). Furthermore, accreditation may not be an option for all sources of high-quality health information. Therefore, accreditation is an imperfect proxy for the evaluation of an organization’s credibility as a source of health information and should be viewed as a preliminary indicator.

Further, there is tremendous variation in rigor and scope across accreditation programs, so accreditation does not convey adherence to a common standard for credibility. It should also be noted that some accreditation programs offer accreditation to organizations that do not share high-quality health information as defined by the authors. For these reasons, accreditation cannot serve as a comprehensive evaluation of credibility as a source of health information and should be supplemented by other forms of vetting.

Academic Health and Medical Journals

Academic journals are scholarly periodicals that publish research or reports specific to a profession or field of study. Many journals promote their publications through social media; in addition, journalists summarize journal articles and share their topline findings through social media. Academic journals are generally affiliated with educational institutions or professional associations. Although they exercise editorial independence, many are owned by for-profit publishing corporations, such as Elsevier.

Academic journals generally adhere to rigorous processes to ensure scientific excellence and integrity, such as peer review. For example, health and medical journals indexed by the U.S. National Library of Medicine (NLM) through MEDLINE must meet clear standards for “scope and coverage, editorial policies and processes”, scientific and methodological rigor, production and administration, and impact (see Box 3) [20]. A list of all MEDLINE-indexed journals is available from the NLM website [21].

Many journals also follow the Recommendations for the Conduct, Reporting, Editing, and Publication of Scholarly Work in Medical Journals produced by the International Committee of Medical Journal Editors (ICMJE), an independent working group of medical journal editors [g]. Broadly, the recommendations establish standards for authorship, disclosure of financial and nonfinancial COI [h], editorial freedom, protection of research participants, corrections and retractions, and handling of scientific misconduct, among other guidance [22]. ICMJE publishes a list of journals that state that they follow the ICMJE recommendations on its website (ICMJE.org) but cautions that it cannot verify the accuracy or completeness of its list. However, echoing this paper’s position that the pursuit of perfection should not hamper incremental progress, ICMJE further notes that “maintenance of such a list may help to promote improvements in the quality of medical science and its reporting by indicating the standards many editors indicate they work to uphold” [23].

Credibility Concerns

Despite rigorous editorial and scientific review processes, journals sometimes allow the publication of articles that contain errors or misinformation, some of which may be retracted later. A social media user who consumes information from such articles may never learn of their retraction. Furthermore, errors or oversimplifications may be inadvertently introduced by entities who communicate about research published by a journal or other source—an issue that is of particular concern in social media, given the brief and ephemeral nature of social media content.

Second, as noted earlier in this paper, the state of science is always evolving. Many academic journals publish the results of unique experiments that have not been—and in some cases cannot be—reproduced by other scientists. This phenomenon is referred to as a lack of “reproducibility” (defined as consistent results using the same data and processes) or “replicability” (defined as consistent answers to the same scientific question, using different data and processes) [24]. However, the scientific community does not always view challenges in reproducibility or replicability as a problem. Rather, early hypotheses and approaches are built on by other scientists and become part of the evolution of knowledge. However, these nuances are not likely to be clear to the average consumer of health information, who may interpret the results of a single experiment as conclusive. Regardless of reproducibility and replicability, entities that consume and communicate about research may place too much weight on a single study that may prove to be an outlier in metaanalyses or literature reviews.

Finally, there are a growing number of journals that misleadingly claim to be high-quality health and medical journals and do not conform with publishing best practices, including those outlined in the previous section. These journals publish almost every submission they receive, charge authors for the publication of their articles, and do not use scientific peer review processes. Further, some are deliberately deceptive by using titles that are similar to established, reputable journals [25]. Prospective authors can perform their own research to determine whether a journal they are considering for publication of their articles is reputable. However, this step is not likely to be taken by someone who encounters information from a journal that seems reputable in social media. Furthermore, for the reasons outlined in this section, the simple inclusion of an article in MEDLINE does not guarantee the absence of errors or misinformation.

Together, these credibility concerns call for an aggressive and sustained effort to educate consumers of health information about the nature of scientific experimentation, quality assurance processes upheld by academic journals, and, generally, the factors by which they can judge source credibility and information quality. The need for such an effort is discussed further in the following section.

Government Organizations

Federal, state, and local government organizations operate under a number of provisions that support their credibility as sources of health information—primarily in the areas of transparency and accountability. As part of the system of checks and balances built into the U.S. government, the Constitution gives the legislative and judiciary branches oversight over the executive branch, where many federal organizations that serve as sources of health information reside (see Box 7). For example, Congress could compel the disclosure of documents or require the sworn testimony of representatives from the CDC to evaluate the agency’s response to COVID-19. Many federal agencies produce information at the request of Congress, with internal and external review to increase the quality of that information. For example, the U.S. Government Accountability Office produces nonpartisan fact-based information at the request of congressional committees and subcommittees and operates a hotline for the public to report on government fraud, waste, or abuse.

Box 7. Major Federal Government Organizations That Serve as Sources of Health Information.

• U.S. Department of Health and Human Services

  • Administration for Children and Families

  • Administration for Community Living

  • Agency for Healthcare Research and Quality

  • Agency for Toxic Substances and Disease Registry

  • Centers for Disease Control and Prevention

  • Centers for Medicare & Medicaid Services

  • Food and Drug Administration

  • Health Resources and Services Administration

  • Indian Health Service

  • National Institutes of Health

  • Office of the Assistant Secretary for Planning and Evaluation

  • Office of the Assistant Secretary for Preparedness and Response

  • Office of the National Coordinator for Health Information Technology

  • Office of the Surgeon General

  • Substance Abuse and Mental Health Services Administration

• U.S. Department of Veterans Affairs

  • Veterans Health Administration

• U.S. Department of Defense

  • Military Health System

• U.S. Department of Homeland Security

  • Federal Emergency Management Agency

• U.S. Department of Agriculture

• White House

  • Office of Science and Technology Policy

• Social Security Administration

• U.S. Agency for International Development

• U.S. Environmental Protection Agency

• U.S. Geological Survey

• U.S. Department of Commerce

  • National Oceanic and Atmospheric Administration

Box 6. The Rise of “Preprint” Scientific Articles During the COVID-19 Pandemic.

“Preprint” refers to an article that has not yet been accepted for publication by an academic journal or undergone peer review. Authors upload preprints to servers designed for that purpose, where they can be accessed by fellow researchers and members of the public. This allows research to be shared, commented on, and refined more quickly than the traditional journal publication process. Some preprints are submitted for publication by a journal at the same time as their upload to a preprint server, and the majority of preprints go on to be formally published by journals [a]. The major preprint server for biomedical research is called medRxiv and was founded by Cold Spring Harbor Laboratory, BMJ, and Yale University. Disclaimer language on the server specifies that, “Preprints are preliminary reports of work that have not been certified by peer review. They should not be relied on to guide clinical practice or health-related behavior and should not be reported in news media as established information” [b].

The scientific community heavily utilized the preprint mechanism to quickly share research and information during the COVID-19 pandemic. Of more than 125,000 scientific articles on COVID-19 during the first 10 months of the crisis, more than 30,000—or about 25 percent of the research—were hosted on preprint servers. According to a study of medRxiv and bioRxiv (a related server for preprint biology research), 85 percent of authors who submitted articles on COVID-19 used a preprint server for the first time during the pandemic. COVID-19 preprints were also widely cited compared with articles on other subjects (almost 60 percent were cited at least once) and shared, including through social media (the most popular preprint was mentioned nearly 27,000 times on Twitter). Despite the servers’ admonition that preprints should not be relied on by the media, nearly 29 percent of COVID-19 preprints were featured in at least a single news article [a].

The increase in usage and popularity of preprints during the COVID-19 pandemic adds a layer of complexity to the discussion of academic journals as credible sources of health information, given the ease with which preprint research may be confused with articles that have undergone formal peer review and editorial oversight. However, preprint servers such as medRxiv (and, by extension, articles that carry a preprint citation) do not follow all of the authors’ principles and criteria for credible sources of health information outlined in this paper.

Public transparency laws reinforce this accountability. For example, the Freedom of Information Act requires disclosure, upon requests from the public, of documents controlled by the federal government (with specified exceptions), which allows for independent scrutiny and critique of government information. The Federal Funding Accountability and Transparency Act requires the disclosure of information about any organization that receives federal funds, while the Digital Accountability and Transparency Act sets standards for availability of this information on the federal website USASpending.gov. The Federal Advisory Committee Act requires that advisory committees to the federal government operate in an open and transparent manner so that the public can access and review the information these groups generate. Government agencies that receive a certain level of research funding (internal or external) must make the results of the research (such as in journal articles) available to the public for free. In addition, many external groups monitor information released by the government.

Similar provisions enhance the credibility of state and local governments as sources of health information, although such provisions may be less strict and may vary from one jurisdiction to another. Furthermore, state lawmakers can support highly divergent public health policies, as demonstrated by variations in mask-wearing guidance during the COVID-19 pandemic [26].

Credibility Concerns

Trust in the federal government is low among some groups, compromising its influence as a credible source of health information. According to a survey from the Pew Research Center, only 24 percent of Americans trust the federal government “to do what is right just about always or most of the time” [27] (this statistic refers to the federal government in general, not as a source of health information). Trends in trust of the government vary among political party affiliation as well as race and ethnicity, suggesting that perception of government credibility may vary across population groups [28].

Furthermore, provisions for transparency and accountability are important attributes of credible sources but are not a guarantee of high-quality information. Recent events have raised further concern, as in the example of the CDC’s removal of scientific guidance after allegations of undue political pressure on the agency’s staff [29]. Clearly, the degree of transparency and accountability upheld by the government at all levels is influenced by the leaders currently in office. Politicization can bias or limit the information that government organizations release. Government organizations may also withhold complete information because of privacy or security concerns, which may result in fragmented or distorted perceptions of issues. Therefore, although the authors believe that government organizations can generally be treated as credible sources, the principles and attributes identified in this paper should apply to them as well, and gaps in credibility should be further examined.

Nonprofit Organizations Not Subject to Standardized Vetting Mechanisms

Many categories of nonprofit organizations that are not subject to standardized vetting mechanisms serve as sources of health information. Some adhere to rigorous standards that align with the principles and attributes outlined in this paper, and some do not. There is no pre-existing, standardized mechanism for evaluating the credibility of sources in this category (although individual mechanisms exist). Therefore, SMPs that wish to assess the credibility of such sources should develop a standardized process for assessing alignment with the principles and attributes identified in this paper.

Table 2 lists types of organizations that share health information (excluding health care organizations, health plans, government organizations, and public health departments), along with the authors’ general observations about the credibility of organizations in each category, drawing from the principles and attributes.

TABLE 2. Other Nonprofit Entities That Share Health Information (Excluding Health Care Organizations, Health Plans, Government Organizations, and Public Health Departments).

Organization Type	Definition	Credibility Observations [a]
Independent organizations or advisory panels that create evidence-based guidance (e.g., “blue ribbon” panels	Entities in this category produce evidence-based conclusions or recommendations at the request of the government or other entities to inform the development of public or organizational policy and practice. An example of an organization in this category is the U.S. Preventive Services Task Force.	These entities generally synthesize information from multiple sources and incorporate consensus processes, peer review, and measures to address bias and conflict of interest. These organizations typically do not engage in lobbying or advertising and maintain strict independence from funding organizations.
Professional associations or societies [b]	Organizations in this category exist to advance the interests of a given field through development of professional standards, supportive policies, and research, among other functions. Most have paying members. An example of an organization in this category is the American Public Health Association.	Many professional associations and societies engage in research or analysis that generally meets the standards for the creation, review, and presentation of scientific content. These organizations also tend to follow rigorous process to maintain transparency and accountability to their members and others in their field. However, many engage in advocacy or lobbying activities on behalf of member interests or mission-specific issues. Credibility assessments should ensure these activities are disclosed and kept separate from the presentation of relevant health information.
Advisory organizations or think tanks	Organizations in this category employ experts and researchers in order to comprehensively monitor and provide opinions and guidance on a given subject or group of subjects. Opinions and guidance are given in the form of media interviews, speeches, news articles, journal articles, books and reports, and beyond. Some may use consensus or peer review processes. An example of an organization in this category is the RAND Corporation.	Many of these organizations engage in research or analysis that generally meets the standards for the creation, review, and presentation of scientific content. However, many think tanks have political biases. Further, many employ scholars or experts who share personal opinions without content oversight from the organization. Credibility assessments should ensure these activities are disclosed and strive to separate ideological messages from relevant health information, as well as examine the knowledge generation processes of these organizations.
Health industry groups	Organizations in this category exist to advance the interest of a given health industry through development of standards, supportive policies, and research, among other functions. Most have paying members. An example of an organization in this category is the American Hospital Association.	Many health industry groups engage in research or analysis that generally meets the standards for the creation, review, and presentation of scientific content. However, many engage in advocacy or lobbying activities on behalf of member interests or mission-specific issues. Although health industry groups may be nonprofit, they represent for-profit interests. Credibility assessments should ensure these activities are disclosed and that profit motives are kept separate from the presentation of relevant health information.
Nongovernmental organizations (NGOs)	NGOs are mission-driven organizations that seek to advance social, political, or humanitarian goals domestically and globally. Many rely on donations. An example of an organization in this category is Partners in Health.	Some NGOs engage in advocacy or lobbying on behalf of mission-specific goals. Some seek to advance political or religious ideologies. Credibility assessments should ensure these activities are disclosed and kept separate from the presentation of relevant health information.
Foundations	Organizations in this category provide funding (through grants or gifts) and other forms of support for nonprofit organizations to advance common goals within the foundation’s area of interest. An example of an organization in this category is the Robert Wood Johnson Foundation.	Foundations may fund and share research and analysis that generally meet the standards for the creation, review, and presentation of scientific content. These organizations usually require rigorous planning and evaluation of the projects they choose to support and generally provide some degree of public transparency in their investments. Foundations may incorporate ideological messages, which should be kept separate from the presentation of relevant health information.
Patient or disease advocacy groups	Organizations in this category exist to advance the interests of patients and their loved ones, generally or in relation to a specific disease or condition. They do so through raising awareness, investing in research, and providing direct support to patients and families, among other functions. Many rely on donations. An example of an organization in this category is the Cystic Fibrosis Foundation.	These organizations may fund and share research and analysis that generally meet the standards for the creation, review, and presentation of scientific content. However, many patient or disease groups engage in advocacy or lobbying activities. Credibility assessments should ensure these activities are disclosed and kept separate from the presentation of relevant health information.
Community health organizations	Organizations in this category exist to advance the health of a given community by raising awareness, fostering engagement, and connecting community members with resources, among other functions. In many cases, these groups focus on culturally competent communication and involve community members in planning and decision making. An example of an organization in this category is DC Health Matters.	Community health organizations may command a high degree of trust among their constituents and therefore serve as important sources of health information. However, there is tremendous variation among these organizations, requiring a high degree of granularity in credibility assessments.
News organizations	Nonprofit news organizations [c] include independent newspapers, magazines, websites, radio or audio networks, and more. An example of an organization in this category is National Public Radio.	News organizations generally use citations and date labels, post corrections and allow for public comment opportunities, exercise editorial independence from funders, and more. However, credibility assessments should keep in mind that many news organizations have political biases and may prioritize attention-grabbing stories over the objective presentation of science-based information.

^[a]Observations in this column consider the authors’ principles and credibility attributes in light of the general practices and characteristics of selected organization types.

^[b]Note that some political advocacy groups have names similar to those of professional associations, such as the Association of American Physicians and Surgeons.

^[c]Nonprofit news organizations are specified in accordance with the scope of this paper, described in section “Scope”. However, for-profit news organizations (and other for-profit entities) are also influential sources of health information and can be difficult to distinguish from nonprofit sources. The credibility attributes and observations laid out in this paper may also be useful in assessing the credibility of for-profit news organizations, as well as other for-profit sources.

Credibility Assessment Steps

To assess the credibility of sources not subject to preexisting, standardized vetting mechanisms that align with the authors’ principles and attributes, SMPs would need to collect and evaluate a standardized set of data. The means of data collection could be either primary or secondary (i.e., SMPs could undertake their own discovery process or rely on information provided by a source). For example, primary data collection might mean using technology to “crawl” a source’s website for evidence of citations, peer review processes, COI disclosures, etc. Secondary data collection might take the form of a credibility attributes and disclosures section that a source could provide to an SMP and post publicly on the homepage of its social media channel(s). This latter approach would require sources to self-regulate and comply with an informal “honor system” or “code of ethics.” For example, a source would have to decide whether the content of an advertisement posted alongside health information constitutes a conflict of interest that could compromise the quality of that information.

On the other hand, as previously mentioned, sources that are subject to such standardized vetting mechanisms can be afforded a preliminary assumption of credibility, as well as government organizations by virtue of their strict accountability practices. However, even sources in these groups should strive to display a preponderance of the authors’ credibility attributes and publicly disclose any deviations (as well as be subject to parallel content evaluation, as described in the following).

For any source type, SMPs’ approach to credibility assessment should include a human-led quality assurance (QA) program. Algorithms and other automated technologies are likely not capable of evaluating every nuance of the credibility attributes. The QA system should verify alignment with source credibility attributes as well as the quality of the information shared. To ensure that consumers are accessing high-quality health information, some form of content assessment is essential as a supplement to source assessment. Although this paper does not provide guidance on principles or mechanisms for content assessment, the authors urge SMPs to invest in research and analysis to quickly build capacity in this area. Priority should be given to high-volume, highly influential sources of health information.

A particularly promising possibility for consideration by SMPs would be to outsource QA functions to an independent third party, either pre-existing or created for this purpose. This approach would bolster the objectivity and integrity of the process by reducing the role of SMPs, which are unavoidably conflicted by their financial and political stake in the performance of the system. To maximize independence and objectivity, this third party organization should not be solely funded by a single SMP.

Identifying Credible Sources for Consumers

Given the preliminary nature of the approach described in this paper and lack of insight into the policies and capabilities of SMPs, the authors do not offer an opinion as to whether or how SMPs should “elevate” credible sources of health information (e.g., via an algorithm). However, when it comes to options for identifying such sources for consumers, the authors suggest that SMPs do not at this time affix a formal label such as “credible” or “non-credible.” Instead, SMPs should provide contextual information that may serve as a “credibility cue” but stop short of a formal designation of credibility. For example, a platform could identify a source as an “Accredited Health Care Organization” and provide a link to a definition of that organization type, including general credibility observations such as those in Table 1. Of note, Twitter has taken a similar approach to identify government- and state-affiliated accounts on its platform [30]. This method would also align with calls to preserve the right of individuals to make independent judgments about the information they are consuming (although it should be acknowledged that consumers’ judgments are impacted by their degree of health and digital literacy, as described in the following). Regardless of the specific approach to source identification, SMPs should be transparent with consumers about their policies as well as the principles and criteria that underlie them. Data collection and assessment, whether primary or secondary, as well as QA activities, should be performed at regular intervals (e.g., semiannually) to ensure sources’ ongoing alignment with the principles and attributes. Potentially, QA activities could also incorporate consumer feedback. Sources that do not demonstrate ongoing adherence should lose any public signal of credibility, and that loss should be made visible to consumers. SMPs may have or could develop lists of sources that are known proponents of harmful information, and these should be made public for the benefit of consumers.

FIGURE 1. Assessment Flowchart for Credibility of Sources of Health Information in Social Media.

[a] This chart is developed for credibility assessment of nonprofit and government organizations only. For-profit companies and individuals that serve as sources of health information should also undergo separate credibility assessment processes.

[b] Pre-existing, standardized vetting mechanisms that align with the authors’ principles and attributes include accreditation, academic journal indexing, and government accountability rules. Even sources subject to one of these mechanisms should strive to meet the authors’ stated credibility principles and attributes.

[c] See Table 1 for a list of principles and credibility attributes.

[d] Ideally, a quality assurance system that includes content assessment should supplement assessment of source credibility.

SMPs should also monitor the policies of peer social media companies, both to be aware of how their own content may be repackaged on other platforms (i.e., perhaps stripped of important contextual information) and to learn, share, and reach alignment on approaches to common challenges.

Parallel Strategies to Supplement Source Assessment

SMPs’ approaches to source assessment should continue to be refined and improved, and, importantly, should be supplemented by other strategies (as called for in public comments; see Box 2).

Content Assessment

Once again, although a reasonable starting point, evaluation of source credibility alone is not an adequate tool to ensure social media users’ access to high-quality health information. SMPs must supplement source assessment strategies with an equally robust system for content evaluation. In the interim, SMPs should clearly explain the limitations of source credibility to consumers (i.e., a source deemed credible is likely to share high-quality information, but not guaranteed). Ultimately, source credibility and information quality should be integrated under a single, streamlined assessment system to maximize clarity and usability for both sources and consumers.

Management of Misinformation

The elevation of credible sources of health information, while an important contribution, is not enough on its own to counteract the harms of misinformation and disinformation. SMPs should maintain parallel strategies to address such false and inaccurate information, as well as sources that deliberately promulgate such information. Admittedly, management of misinformation is a highly complex challenge, both politically and legally, making the elevation of credible sources and high-quality information a potentially more feasible priority.

As noted previously, the state of science and knowledge is always evolving, and information that was once consistent with the best available evidence at the time can quickly become outdated. Credible sources can avoid the perception of misinformation by using clear date labels and striving to update content regularly.

Health Literacy, Culturally Competent Communication, and Community Relationships

Regardless of the eventual system for elevating credible sources and high-quality information, consumers will still make their own judgments about which sources and information to trust. In fact, one of the major themes from the public comment period, summarized in Box 2, was that SMPs must protect freedom of speech and the autonomy of users in accessing the information that they choose.

Users seeking health information may not be satisfied with an SMP’s assessment of source credibility or information quality. As explained by Lisa Fitzpatrick, founder and CEO of the Washington, DC-based community organization Grapevine Health, people are resourceful and often consult many sources, both online and offline, before reaching a conclusion [31]. Resourcefulness is an asset if people are empowered and provided with ready access to high-quality health information. Although many people have a high degree of health literacy, a large-scale effort is needed to ensure that resources are in place to support and educate all people to become savvy, informed, and science-literate users of social media. This concept is an important aspect of information equity [i] (across literacy levels, preferred languages, location/locality, etc.)—and, by extension, health equity.

Therefore, SMPs should invest in evidence-based health literacy and consumer education strategies to support the success of their in-house approaches to elevating credible sources and high-quality information. Such strategies could be designed and executed by the platforms themselves, but a better approach may be to delegate to independent third parties.

Consumers’ evaluation of the credibility of online information goes beyond source and content characteristics to considerations of design, or the way information is presented [3]. A source’s credibility is of little relevance if it fails to connect with its audience. As an illustration, Fitzpatrick shared a quote from a community member: “I don’t understand what doctors are saying, and if I don’t understand you, I can’t trust you” [31]. Several of the public comments the authors received echoed this point, noting that credible sources may not always present information in a manner that is appealing, engaging, or culturally competent.

“Cultural competence” refers to the ability to interact effectively with diverse audiences by recognizing and responding to variations in social, cultural, and linguistic needs [32]. Diversity should be considered across multiple dimensions, including but not limited to, race/ethnicity, education level, socioeconomic status, age, and political affiliation. Political affiliation may be particularly relevant to this discussion, as there is a 22 percent difference between Republicans who express a great deal of confidence in medical scientists (31 percent) and Democrats (53 percent) [33]. Republicans are also less approving of SMPs’ efforts with regard to content moderation, making it important to engage with bipartisan opinions in the design of policies such as those discussed in this paper [34].

Organizations that strive to be credible sources of health information should invest in strategies to improve their communication skills by using language and images that are informed by cultural contexts as well as understandable and engaging (at the same time, however, they must share complete and precise information and avoid oversimplification) [j]. They should also seek to build authentic, collaborative relationships at the community level. These efforts may help to foster consumers’ trust— which is all the more important as communication challenges during the COVID-19 pandemic may have impacted the perceived credibility of public health authorities in the United States [35].

As with efforts to advance health literacy and consumer education, SMPs should consider financial support for such endeavors to promote competent communication by credible sources on their platforms. Social scientists in the fields of science, risk, and health communication would make productive partners for SMPs in this work.

Source Self-Regulation and Consumer Evaluation

Rather than waiting for evaluation by SMPs, sources of health information that wish to be considered credible should take proactive steps to apply science-based, objective, and transparent and accountable principles to their institutional practices and presentation of information. The principles and attributes set forth in this paper may provide a useful starting point, as well as other resources (including those described in Box 3 and Appendix A). By making these efforts transparent and highly visible to the public and their peers, organizations can begin to build a self-sustaining, cultural “norm” for credibility that may ultimately increase the quantity of high-quality information in social media and have a cascading effect on the quality of information both online and offline.

A transparent set of principles used by sources to assert their own credibility would also support consumers’ independent judgments, as called for in the public comments that informed this paper. Consumer evaluation can supplement the efforts of SMPs to identify and elevate credible sources and high-quality information, as well as ensure accountability when sources fail to adhere to standards they set for themselves. However, there are several major limitations to the application of consumer evaluation in this context.

First, as noted previously, consumers have differing levels of health and digital literacy, which impacts their ability to assess the credibility of a source or the quality of a piece of information. This is why SMPs whose goal is to increase access to high-quality information should support efforts to advance health literacy. Second, consumer opinion is often captured through measurements of source or content popularity, comments, and recommendations, among other means [3]. Such elements can be easily “hijacked” by entities whose goal is to spread misinformation or disinformation, as has been seen in anti-vaccination disinformation campaigners’ use of “bots” and “trolls” to amplify their reach and engagement on Twitter [37]. Finally, as noted in Box 4, the interplay between consumer trust and credibility is complex and may be difficult to parse in consumer feedback. Nevertheless, consumer feedback is an important consideration, including in the design of source and content evaluation policies.

Box 4. The Relationship Between Trust and Credibility.

Trusted is not synonymous with credible. Sources considered credible by the authors’ definition may not be trusted by all individuals and groups, while sources that are widely trusted may not be credible. However, trust affects the perception of credibility, and by extension, the influence of credible sources of health information. For example, according to a survey by the RAND Corporation, trust in the Centers for Disease Control and Prevention (CDC) declined by about 10 percent during the COVID-19 pandemic. The authors of the survey suggest that “public trust in federal government agencies has never been as important as it has been during the COVID-19 pandemic, yet public suspicions of scientific experts and levels of distrust of government institutions are increasing” [a]. Because the CDC plays an essential role in disseminating scientific information and public health guidance to the nation, its trust deficit is a significant problem that could have bearing on health outcomes.

Therefore, no matter how credible public health organizations might be, building trust should be a primary focus of their approach to information sharing and relationship building (through social media as well as other means). In a 2021 rapid expert consultation on promoting trust around the COVID-19 vaccine, the National Academies of Sciences, Engineering, and Medicine offered strategies for engaging communities to combat mistrust, including forming partnerships with community organizations; “engaging trusted messengers who have roots in the community”; and communicating across multiple, accessible channels [b].

Just as building trust should be a priority for credible sources of health information, ensuring the credibility of highly trusted and influential sources should be of primary concern for SMPs and other stakeholders concerned with public health and health communication.

Ethical and Public Health Considerations

In an effort to ensure that the guidance in this paper increases access to high-quality health information, thereby promoting health, and minimizes harm, the authors have also outlined ethical and public health considerations for the approach described herein.

Ethical Considerations

Control of Information

Control of information emerged as a key concern during the public comment period. SMPs’ efforts to increase access to high-quality information (by elevating credible sources) may be perceived as censorship or an attempt to limit the autonomy of information consumers (see Box 2). Platforms should strive to engage consumers in the design and evaluation of such strategies and maintain public transparency around policies and actions taken. Policies should balance the need to minimize the harm that could occur through propagation of health misinformation (as seen during the CO-VID-19 pandemic [7]) with the right of the consumer to undertake a personal evaluation and judgment.

Structural Bias

BIPOC have historically been underrepresented in many institutions that enjoy a reputation for credibility today. For example, racial segregation persisted in U.S. universities until the latter half of the 20th century, and more insidious forms of prejudice endure to this day [37]. BIPOC researchers and their research insights are underrepresented in clinical and biomedical fields and are less likely to receive federal funding than their White counterparts [38]. According to 2020 data from the U.S. Office of Personnel Management, Black and other people of color made up 38 percent of the federal workforce but only 22 percent of Senior Executive Service positions [39]. This historic and ongoing under-representation of BIPOC and others in positions of influence in academia, science, health care, and government means information shared by these institutions may not always reflect the experiences of or resonate with these groups— thereby perpetuating underrepresentation and information inequity.

To counter this bias and increase equity and representation, SMPs should make a concerted effort to identify and promote sources that are not only credible but also trusted and utilized by diverse audiences, including BIPOC and other groups, such as new immigrants, LGBTQIA+ individuals, religious minorities, and people with disabilities. SMPs should assess consumer data to identify sources that are heavily utilized by marginalized groups and prioritize them for credibility evaluation and potential elevation. Partnerships with groups that represent the rights and health of such groups will be essential to the success of this effort.

Financial Conflict

A system that elevates credible sources of health information may create a new “credibility brand” that is profitable for both sources and SMPs themselves. As set forth in the authors’ principles and criteria, credible sources should take steps to ensure that financial and ideological interests do not compromise the presentation of science-based health information. However, financial gain and enhanced influence may be unavoidable collateral effects of designation as a credible source in social media channels.

SMPs should support research to understand the impact of credibility designations on the quality of information shared by sources, on sources’ level of influence both inside and outside social media, and on sources’ financial status. Advertisements should not be attached to high-quality health information shared through SMPs, both to minimize financial conflicts of interest and to avoid compromising the quality and accessibility of the information (e.g., with distracting and potentially misleading ads).

To uphold their integrity, SMPs should separate their own profit motives as much as possible from efforts to elevate credible sources of health information. One way to achieve this would be for platforms to work with independent third parties to design and implement source and content evaluation and moderation strategies.

Feasibility and Appropriateness of SMPs’ Role

This paper has made significant asks of SMPs beyond the initial task of identifying and elevating credible sources of health information. In addition to a quality assurance system for source evaluation, these include parallel strategies for content assessment and misinformation management, as well as collaborative efforts to promote equity and support public health research. Such activities will require a considerable investment of time and resources, and SMPs do not necessarily have a financial incentive to make this investment [41]. Some argue that platforms are actually disincentivized from interventions that could dampen profits driven by advertising and high rates of engagement with misinformation [42]. However, SMPs’ engagement in highly visible social responsibility efforts, as well as an enhanced corpus of credible sources that are trusted and relied on by consumers, may prove financially and politically beneficial.

The complexity of SMPs’ interests merits careful consideration of their role as moderators of health and other crucial public information. Although the authors believe that SMPs should take a proactive role across several dimensions, as outlined in this paper, government regulation and delegation to independent third parties should also be considered as potential supplementary approaches.

Public Health Considerations

Health Equity

A system to elevate credible sources—and thereby increase access to high-quality health information—must be designed to support health equity, as well as information equity, and not cement existing inequities. Diversity and inclusion are important components of a system that promotes health equity. As noted previously in the “Structural Bias” section, efforts to eliminate racial bias and foster diverse representation among credible sources of health information are important to avoid perpetuating health inequities.

Older people, adolescents, people with lower education and income levels, and racial and ethnic minority groups are more likely to face challenges related to limited health literacy [43]. Therefore, efforts to foster health literacy, engage in culturally competent communication, and build and sustain community relationships and trusted networks—as called for by the authors as a supplemental strategy to elevating credible sources—are supportive of health equity. SMPs should be sure to use accessible language when defining and explaining policies related to credible sources of information. SMPs should also consider digital literacy and strategies to address equity in access to high-quality digital information—a challenge referred to as the “digital divide.”

The digital divide is defined as “disparities in technology access and use [that have] compounding effects on existing inequities along income, educational, racial, and geographic dimensions” [44]. Although approximately three-quarters of Americans have access to high-speed broadband internet at home, rates vary significantly by education level and income. In 2019, only 46 percent of people with less than a high school education had broadband, compared with 93 percent of college graduates. In 2021, the rates were 57 percent for people making less than $30,000 annually and 92 percent for people making more than $75,000. Disparities by race and ethnicity are less dramatic but still significant: 80, 71, and 65 percent for white, Black, and Hispanic people, respectively, in 2021 [44]. Despite this, Black and Hispanic people are more deeply engaged in social media than whites across some dimensions [45].

The digital divide is an important consideration for SMPs as well as other platforms that facilitate the sharing of health information. If efforts to increase access to high-quality health information disproportionately benefit highly educated, wealthy, and white people, then they are cementing health and information inequities.

Contribution to Public Health Research

SMPs can be important partners in improving public health, but only if they agree to share data (e.g. back-end data, algorithms and use engagement metrics, content moderation processes) with researchers. This paper provides guidance that is intended to increase access to high-quality health information and thereby promote individual and population health. However, SMPs alone have access to data that could form the basis of important health and behavioral research about how policies such as those discussed herein would actually affect the consumption of high-quality health information, as well as whether enhanced access to such information would favorably impact offline outcomes.

In addition to sharing such data as outlined previously, SMPs should be transparent about the methods they use to promote consumption of high-quality health information (e.g., through algorithmic recommendations), as well as the full scope of their policies and processes with regard to health information of any quality. As noted earlier, health misinformation and disinformation spread through social media can negatively impact health outcomes, and SMPs should take responsibility for and develop solutions to mitigate elements of their systems that enable such information to flourish.

SMPs’ reluctance or failure to share such data and moderation methods would prevent fully productive collaborations with the public health and behavioral science communities. To be considered credible themselves, platforms should make a public and highly visible commitment to transparency and accountability, especially with regard to data, policies and methods that could impact public health.

Conclusion

Increasing access to high-quality health information in social media is a complex challenge that requires navigating tremendous volume and variation among sources and information; the continuous evolution of science and knowledge; and significant ethical quandaries—chief among them, the need to protect free speech and consumers’ right to autonomy while minimizing the risk of harm from misinformation. To date, attempts at social media content moderation have been met with controversy and calls for federal regulation from both sides of the aisle [47]. Nevertheless, the potential influence of health information shared through social media on health outcomes, at both an individual and population level, compels action, even with the knowledge that such action will be incomplete at first.

This paper has presented guidance that could be leveraged by SMPs in identifying credible sources of health information—an incremental step toward the goal of enhancing access to high-quality health information. Although the scope of this discussion has been limited to U.S.-based nonprofit or government sources only, it is likely that many of the principles, attributes, and considerations can be applied to for-profit sources or individuals, as well as sources outside the United States. Efforts to fully assess the credibility of these sources, many of which are highly influential, should be an urgent priority for SMPs.

However, source evaluation is not a comprehensive solution. Several parallel strategies are required to ensure information quality and combat the risks of health misinformation, as detailed earlier. Foremost among these is a strategy to assess information quality and develop content moderation plans in response. The authors acknowledge the infeasibility of evaluating the accuracy and balance of every piece of health information on social media. However, a system of “spot checks” for quality and integrity, supported by machine learning technology but supplemented by expert human evaluation, is within reach. SMPs should invest in developing principles, guidelines, and applications for content assessment alongside strategies for source evaluation. Ultimately, the two approaches should be consolidated in a single system for the identification and elevation of high-quality health information. As previously noted, SMPs’ efforts in these areas should be supplemented by government regulation or delegation to independent third parties.

SMPs cannot, and should not, tackle this challenge alone. As those ultimately impacted by social media source or content curation strategies, consumers must be engaged in developing such strategies. Public engagement is also essential to promote transparency, foster trust, and minimize perceptions of censorship or paternalism. Organizations that use social media to share information have an important role as well, and should hold themselves publicly accountable to a set of principles that supports the quality of the information they share, as well as their own institutional credibility. Together, the actions taken by consumers, organizations, and SMPs can move toward greater availability and accessibility of high-quality health information.

Finally, consumers and organizations that utilize social media deserve to understand the mechanics and the outcomes of policies that affect the information they receive and share. Therefore, SMPs should make their source and content moderation practices (e.g., algorithms) and relevant data accessible to independent behavioral and public health researchers to analyze the effects on information consumption as well as offline behaviors. Without such information, consumers and organizations that collaborate with SMPs will have no way of knowing whether policies are justified or effective. To be effective partners in improving health, SMPs must make a firm commitment to transparency and accountability.

APPENDIX A. Models for Assessment of Source Credibility

National Library of Medicine

The National Library of Medicine (NLM) has developed at least three major source evaluation systems that provide useful examples for the task at hand: MEDLINE indexing, MedlinePlus indexing, and the Disaster Lit database.

MEDLINE Journal Selection

The National Library of Medicine (NLM), the world’s largest medical library, uses stringent criteria to determine whether a journal should be included in MEDLINE, NLM’s premier bibliographic database. MEDLINE is the primary component of PubMed, a freely accessible online literature database developed and maintained by the NLM National Center for Biotechnology Information (NCBI), with new citations added daily.

In considering whether a journal merits inclusion in MEDLINE, NLM considers “the scientific and editorial character and quality of a journal”. Several factors are used in making this decision, including the scientific policy set by the NLM Board of Regents [1], the suitability of the journal for the NLM Collection, and the recommendations of an NIH Federal Advisory Committee, the Literature Selection Technical Review Committee (LSTRC) [2].

The LSTRC evaluates journals for MEDLINE based on 5 critical elements, including scope and coverage, editorial processes, scientific rigor/methodological rigor, production and administration, and impact (more detail on all characteristics can be found at https://www.nlm.nih.gov/medline/medline_journal_selection.html [3]).

By using such clear and detailed guidelines for their journal selection process, NLM’s goal with MEDLINE is to demonstrate a high level of scientific rigor while reflecting sufficient geographic heterogeneity.

MEDLINEPlus.

MedlinePlus is another resource created and curated by the NLM. Unlike MEDLINE, which is primarily used by researchers and professionals, MedlinePlus is a health information website for patients, their families and friends, and the general public. MedlinePlus aggregates health information from a variety of sources. “Some pages, like the medical encyclopedia and drug information, are licensed from outside sources. Other pages, like health topic pages, medical tests, and genetic pages, are created specifically for MedlinePlus” [4].

Information on MedlinePlus is reviewed and updated according to guidelines developed for each type of page. Health topics are “updated as new information becomes available, and broken links are checked and fixed daily” [4]. Medical tests are reviewed at least every 3 years, although content is also updated as needed between review cycles. Genetics pages are reviewed by experts in the field before being posted to MedlinePlus and with each substantial revision thereafter, and patient support and advocacy groups provide feedback on select content. MedlinePlus also adds new and updated articles and illustrations to the A.D.A.M. Medical Encyclopedia each month, and an A.D.A.M. Editorial Policy ensures that included content presents evidence-based health information.

In determining whether to link to a nongovernment website, MedlinePlus staff considers a range of criteria related to the website’s content, advertising, availability and maintenance, and user privacy. In examining a resource’s content, these criteria include:

• The website offers a description of the organization, including a mission statement that aligns with MedlinePlus’s aim to provide accurate health information to patients and their families and friends.

• “The organization provides accurate, science-based information that complements or enhances the health information found on MedlinePlus.

• The source of the content is trustworthy and transparent.

• The primary purpose of the website is educational, and the information is unbiased.

• The information provided is easy to understand, easy to navigate, and well organized.

• The website has original content.

• The website links only to reliable sources that meet MedlinePlus guidelines for links or other clearly stated guidelines established by the website” [5].

In addition to such content considerations, MedlinePlus gives preference to pages with no advertising. If the website has advertising, it must display an advertising policy that clearly separates educational content from advertising or sponsorship. MedlinePlus will not link to web resources that present content suggesting that Medline Plus endorses certain commercial products or services. For a resource to be linked on MedlinePlus, the website must also be consistently available, include contact information for customer support, and provide current information. Furthermore, websites must not require users to register, become a member of the organization, or pay a fee to view health information. Finally, MedlinePlus criteria specify that if a website collects personal information, it must clearly display “a privacy policy that explains how information collected from users remains private and confidential. If a website displays advertising, it should prevent advertisers and sponsors from collecting any personally identifiable information from users” [5].

In addition to these criteria for resources linked to MedlinePlus, for all the pages on MedlinePlus, a “page last reviewed” date is available near the bottom of the page to indicate “when the entire topic was reviewed and updated while a “page last updated” date indicates when any information was added to or removed from the health topic page” [5]. These additional indicators allow users to verify the currency of the content they are consuming.

The CRAP Test

Molly Beestrum, education and curriculum coordinator at Northwestern University’s Galter Health Sciences Library & Learning Center, developed a system known as the CRAP Test that can be used in deciding whether a website is a credible, valid source. The CRAP Test considers four major website attributes: currency, reliability, authority, and purpose. To apply Beestrum’s test, Colorado Community Colleges Online suggests asking the following questions:

“Currency

• How recent is the information?

• How recently has the website been updated?

• Is it current enough for your topic?

Reliability

• What kind of information is included in the resource?

• Is content of the resource primarily opinion? Is it balanced?

• Does the creator provide references or sources for data or quotations?

Authority

• Who is the creator or author?

• What are the credentials? Can you find any information about the author’s background?

• Who is the publisher or sponsor?

• Are they reputable?

• What is the publisher’s interest (if any) in this information?

• Are there advertisements on the website? If so, are they clearly marked?

Purpose

• Is this fact or opinion? Does the author list sources or cite references?

• Is it biased? Does the author seem to be trying to push an agenda or particular side?

• Is the creator/author trying to sell you something? If so, is it clearly stated?” [6]

Clinical Practice Guidelines We Can Trust

The 2011 Institute of Medicine (IOM) study Clinical Practice Guidelines We Can Trust is another resource that may provide useful insights into the determination process for high-quality health information in social media [7]. Clinical practice guidelines (CPGs) help to reduce the level of uncertainty in clinical practice by establishing standards of care backed by strong scientific evidence. These standards “are informed by a systematic review of evidence and assessment of the benefits and costs of alternative care options” [7]. However, many different sets of CPGs have been developed and employed, to varying degrees of success. With nearly 27,000 guidelines in the National Guideline Clearinghouse (NGC) and numerous additional commercial guidelines, it can be challenging to “identify guidelines based on high-quality development methods. Although the NGC provides a standardized summary of each CPG posting, describing its development methodology and evidence base and providing a link to the full guideline, the NGC makes no quality judgment” [7]. As a result, it can be difficult for stakeholders to be confident of CPG quality.

To combat this issue, Clinical Practice Guidelines We Can Trust proposed eight standards for developing trustworthy CPG and called for the development of a mechanism to identify guidelines that meet these standards. These standards include: “emphasizing transparency; management of conflict of interest; systematic review—guideline development intersection; establishing evidence foundations for and rating strength of guideline recommendations; articulation of recommendations; external review; and updating” [7]. The report identifies three options in determining whether a CPG meets these standards: “1) identifying each guideline to see if it meets the specified standards; 2) certifying organizations producing guidelines that comply with quality standards; or 3) acknowledging standards compliance for each guideline production process prior to development of the guideline” [7].

Due to the large number of CPGs, the report suggests certification of organizations with trustworthy CPG development procedures rather than identification of individual trustworthy CPGs or identification of the development process for each CPG. This type of evaluation would entail “reviewing the procedures that applicant organizations use to produce various types of guidance, providing an identifiable mark to be placed on future CPGs of those organizations meeting accreditation requirements, and agreeing to maintain the approved processes during a 3-year accreditation period” [7]. This certification process “would not endorse particular drugs or treatment options for medical conditions or make clinical decisions about the guidelines it reviews” [7]. Instead, it would merely certify the organization’s guideline development process and identify the resulting CPGs as trustworthy.

Health on the Net Foundation Certification

The Health on the Net Foundation (HON) is an international nonprofit organization based in Switzerland. The HON Code of Conduct (HONcode) was created to help standardize the reliability of medical and health information available online by defining a set of rules to hold website developers accountable to basic ethical standards in the presentation of information. This voluntary certification system is based on an “active seal” concept that helps users identify sources of reliable information.

To be certified by HON, a website must formally apply for registration. If accepted, the site must comply with all eight principles enumerated in the HONcode:

• 1.“Authoritative: Any medical or health advice provided and hosted on this site will only be given by medically trained and qualified professionals unless a clear statement is made that a piece of advice offered is from a non-medically qualified individual or organization.
• 2.Complementarity: The information provided on this site is designed to support, not replace, the relationship that exists between a patient/site visitor and his/her existing physician.
• 3.Privacy: Confidentiality of data relating to individual patients and visitors to a medical/health Website, including their identity, is respected by this Website. The Website owners undertake to honor or exceed the legal requirements of medical/health information privacy that apply in the country and state where the Website and mirror sites are located.
• 4.Attribution: Where appropriate, information contained on this site will be supported by clear references to source data and, where possible, have specific HTML links to those data. The date when a clinical page was last modified will be clearly displayed (e.g., at the bottom of the page).
• 5.Justifiability: Any claims relating to the benefits/performance of a specific treatment, commercial product, or service will be supported by appropriate, balanced evidence in the manner outlined previously in Principle 4.
• 6.Transparency: The designers of this Website will seek to provide information in the clearest possible manner and provide contact addresses for visitors that seek further information or support. The webmaster will display his/her E-mail address clearly throughout the Website.
• 7.Financial disclosure: Support for this Website will be clearly identified, including the identities of commercial and non-commercial organizations that have contributed funding, services, or material for the site.
• 8.Advertising policy: If advertising is a source of funding, it will be clearly stated. A brief description of the advertising policy adopted by the Website owners will be displayed on the site. Advertising and other promotional material will be presented to viewers in a manner and context that facilitates differentiation between it and the original material created by the institution operating the site” [8].

To determine whether a site adheres to these standards, sites requesting certification complete an interactive, online questionnaire that tells them what they must add or modify to conform to the HONcode principles. Then an HON team member inspects the site to verify compliance. Once a site has been verified, it identifies itself with the blue and red HONcode hyperlink (or “active”) seal displayed in a prominent location, usually at the bottom of the homepage. Subscribing sites are subject to unannounced checks by HON to ensure continued compliance, and HON also relies on user reports to maintain website reliability.

URAC Health Content Provider and Health Website Certification

URAC provides health care organizations with evidence-based accreditation programs, including health content provider and health website certification. URAC accreditation involves a five-phase, voluntary process that requires reaccreditation every 3 years and is designed to facilitate continuous quality improvement. In the first phase, applicants provide URAC with standard information about their organization as well as specific information related to the type of certification being sought. In the second phase, a lead reviewer evaluates the submitted documents to determine whether they comply with URAC’s standards. During this step, the lead reviewer may provide recommendations to the applicant on how to revise an application to conform to URAC’s policies. The third phase involves an on-site validation review to ensure that the organization is following the standards in practice. In the fourth step, the URAC review team presents an anonymous report to a voluntary accreditation committee. This team of health experts is familiar with URAC standards and determines whether an organization receives full or partial accreditation. The final phase of the accreditation process includes ongoing monitoring such as random surveillance and required reporting of quality measures.

In all certification programs, URAC employs these five phases to focus on risk management, operations infrastructure, performance monitoring and improvement, and consumer protection and empowerment. Certification-specific standards are then designed to further support these goals. For instance, the” Health Content Provider accreditation process examines key areas such as disclosures, health content and service delivery, quality oversight committee, policies and procedures, health content and personal health management, and accountability” [9]. In contrast, considerations that factor into earning a Health Website Accreditation include privacy and security, health content editorial processes, disclosure of financial relationships, website linking policies, and consumer complaint processes.

APPENDIX B. TABLE B-1: Types, Definitions, and Accreditors of Nonprofit Accredited Organizations That May Serve as Sources of Health Information in the United States and What Ac-creditation Signifies

Organization Type	Definition	Accreditors
EDUCATIONAL INSTITUTIONS
What Accreditation Signifies for Organizations in This Category: Educational institutions that earn accreditation have chosen to participate in a voluntary, rigorous process to demonstrate their performance against standards for integrity and continuous improvement; academic excellence; a high level of leadership and employee performance; the well-being of students and patients (as applicable); and appropriate conduct of research.
Schools of medicine	A medical school is a postgraduate educational institution, generally attached to a larger university system, that awards professional degrees to physicians.	Accreditation Council for Continuing Medical Education Accreditation Council for Graduate Medical Education Commission on Osteopathic College Accreditation Council on Podiatric Medical Education Liaison Committee on Medical Education
Schools of nursing	Schools of nursing are undergraduate and postgraduate educational institutions, generally connected to a larger university system, that confer a range of professional or academic nursing degrees—including BSN, MSN, PhD, DNP, and others—as well as continuing education for practicing nurses.	Accreditation Commission for Education in Nursing, Inc. Accreditation Commission for Midwifery Education Commission on Collegiate Nursing Education Council on Accreditation of Nurse Anesthesia Educational Programs National League for Nursing Commission for Nursing Education Accreditation
Schools of public health	Schools of public health are undergraduate and postgraduate educational institutions, generally connected to a larger university system, that confer bachelors, masters-, or doctoral-level academic degrees in public health.	Council on Education for Public Health
Schools of dentistry	Schools of dentistry are postgraduate educational institutions that award degrees to dental professionals.	Commission on Dental Accreditation
Schools of pharmacy	Schools of pharmacy are postgraduate educational institutions that award degrees to pharmacy professionals.	Accreditation Council for Pharmacy Education
Universities	Universities are undergraduate and postgraduate educational institutions that confer academic degrees. Departments or schools of biomedical science, health policy, social science, and others may serve as sources of health information.	Association for the Accreditation of Human Research Protection Programs Higher Learning Commission Middle States Commission on Higher Education New England Association of Schools and Colleges Commission on Institutions of Higher Education Northwest Commission on Colleges and Universities Southern Association of Colleges and Schools Commission on Colleges WASC Senior College and University Commission
HEALTH CARE ORGANIZATIONS
What Accreditation Signifies for Organizations in This Category: Health care organizations that earn accreditation have chosen to participate in a voluntary, rigorous process to demonstrate their performance against standards for patient safety, high-quality care, and continuous improvement processes.
Ambulatory care providers	Ambulatory care is care provided in outpatient settings. Types of ambulatory care organizations include behavioral health centers, birthing centers, dental offices, dialysis centers, medical offices, occupational health centers, outpatient surgical centers, pain management centers, podiatry centers, radiation oncology practices, student health clinics, urgent care practices, women’s health clinics, and more. Ambulatory care covers Federally Qualified Health Centers (which include Community Health Centers, Migrant Health Centers, Health Care for the Homeless, and Health Centers for Residents of Public Housing) [b]. This category also includes Indian Health Service (HIS) and U.S. military and veterans ambulatory health care organizations.	Accreditation Association for Ambulatory Health Care (AAAHC) [a] Accreditation Commission for Health Care (ACHC) [a] The Compliance Team [a] Commission for the Accreditation of Birth Centers Community Health Accreditation Partner (CHAP) [a] Health Facilities Accreditation Program (HFAP) [a] The Joint Commission (TJC) [a] National Dialysis Accreditation Commission [a]
Hospitals	A hospital is a health care institution that provides primarily inpatient services including medical, surgical, or psychiatric treatment [c]. Types of hospitals include academic medical centers, acute care and long-term care hospitals, critical access hospitals, and rehabilitation facilities. There are also a wide variety of specialty hospitals, including cancer, stroke, and cardiac centers; children’s hospitals; psychiatric hospitals; women’s hospitals; and more. This category also includes Indian Health Service (HIS) and U.S. military and veterans hospitals.	Center for Improvement in Healthcare Quality [a] Commission on Accreditation of Rehabilitation Facilities [a] DNV GL Healthcare [a] HFAP TJC
Home-based health care providers	A home-based health care provider offers services for illness or injury in a patient’s home, including wound care, medication administration and management, nutrition counseling, and more. Home-based health care also includes hospice [c] and palliative care.	ACHC CHAP Joint Commission
NONPROFIT HEALTH PLANS
What Accreditation Signifi es for Organizations in This Category: Health plans that earn accreditation have chosen to participate in a voluntary, rigorous process to demonstrate their performance against standards for quality improvement, management, credentialing, and member services and communication, among others.
Nonprofit health plans	Nonprofit health plans provide coverage (insurance) for health and medical expenses and often provide preventive health services.	National Committee for Quality Assurance URAC
PUBLIC HEALTH DEPARTMENTS [e]
What Accreditation Signifies for Organizations in This Category: Public health departments that earn accreditation have chosen to participate in a voluntary, rigorous process to demonstrate their performance against standards for ability to carry out the 10 Essential Public Health Services[f], effective department management, and effective communication with the governing entity (e.g., the state).
Public health departments (state, tribal, local, territorial, and Army Installation)	Public health departments provide services including disease and injury prevention, infectious disease response, and public education and health promotion.	Public Health Accreditation Board

NOTES:

^[a]CMS deeming authority

^[b]See https://www.hrsa.gov/opa/eligibility-and-registration/health-centers/fqhc/index.html

^[c]See https://www.cms.gov/Research-Statistics-Data-and-Systems/Research/ResearchGenInfo/Downloads/DataNav_Glossary_Alpha.pdf.

^[d]Hospice care can also be provided in inpatient settings

^[e]Federal health plans and public health departments are also government organizations, which are subject to additional transparency and accountability rules

^[f]See https://www.cdc.gov/publichealthgateway/publichealthservices/essentialhealthservices.html

APPENDIX C. Questionnaire Used to Collect Public Comments on the “Preliminary Discussion Document” for the National Academy of Medicine Project on “Principles for Defining & Verifying the Authority of Online Providers of Health Information”

Background

In order to enhance the accessibility of trustworthy health information on its platform, YouTube asked the National Academy of Medicine (NAM) to identify preliminary definitions of “authoritative” sources of health information and the criteria by which these sources derive and maintain their authority. The NAM project will also outline ethical and public health considerations for large-scale content curation strategies.

An expert advisory group will gather information and deliberate in order to author a peer-reviewed discussion paper for publication in NAM Perspectives in summer 2021. Papers published in NAM Perspectives are individually authored and do not reflect consensus positions of the NAM, the National Academies, or the authors’ organizations. See project webpage >>

YouTube has informed the NAM that outcomes from this project will help to inform YouTube’s identification and raising of “authoritative” sources of health information, but will not disadvantage sources that do not meet the requirements of specific authoritative source categories. Further, the NAM understands that YouTube will identify and characterize such “authoritative” sources in order to provide context for users but will not confer a formal designation of authority or trustworthiness at the source level. The project will also generate principles that may be of use to online platforms other than YouTube.

Public Discussion Document

To inform the project, the NAM is seeking feedback on a public discussion document containing preliminary principles and questions. The public comment period will last from 12 pm ET on April 5, 2021, until 11:59 pm ET on April 9, 2021.

An anonymized synthesis of feedback received through this process will be posted on the project webpage after the comment period closes. The NAM reserves the right to disregard feedback it considers to be off topic or inappropriate.

Download the discussion document here >>

To leave a comment, please complete the form below.

First and last name of commenter:*

Email address of commenter:*

If you are commenting on behalf of an organization, please enter the organization name here:

One-sentence summary of your comments:*

Complete comments:*

Would you like to receive email updates about this project?*

[ ] Yes

[ ] No

* = Question is required

Funding Statement

The views expressed in this paper are those of the authors and not necessarily of the authors’ organizations, the National Academy of Medicine (NAM), the National Academies of Sciences, Engineering, and Medicine (the National Academies), or the National Institutes of Health (NIH). The paper is intended to help inform and stimulate discussion. It is not a report of the NAM or the National Academies.