Social but safe? Quality and safety of diabetes-related online social networks

Abstract

Objective

To foster informed decision-making about health social networking (SN) by patients and clinicians, the authors evaluated the quality/safety of SN sites' policies and practices.

Design

Multisite structured observation of diabetes-focused SN sites.

Measurements

28 indicators of quality and safety covering: (1) alignment of content with diabetes science and clinical practice recommendations; (2) safety practices for auditing content, supporting transparency and moderation; (3) accessibility of privacy policies and the communication and control of privacy risks; and (4) centralized sharing of member data and member control over sharing.

Results

Quality was variable across n=10 sites: 50% were aligned with diabetes science/clinical practice recommendations with gaps in medical disclaimer use (30% have) and specification of relevant glycosylated hemoglobin levels (0% have). Safety was mixed with gaps in external review approaches (20% used audits and association links) and internal review approaches (70% use moderation). Internal safety review offers limited protection: misinformation about a diabetes ‘cure’ was found on four moderated sites. Of nine sites with advertising, transparency was missing on five; ads for unfounded ‘cures’ were present on three. Technological safety was poor with almost no use of procedures for secure data storage and transmission; only three sites support member controls over personal information. Privacy policies' poor readability impedes risk communication. Only three sites (30%) demonstrated better practice.

Limitations

English-language diabetes sites only.

Conclusion

The quality/safety of diabetes SN is variable. Observed better practice suggests improvement is feasible. Mechanisms for improvement are recommended that engage key stakeholders to balance autonomy, community ownership, conditions for innovation, and consumer protection.

Introduction

Internet engagement centered on health is expanding beyond online search activity, reported for example by 66% of healthy US adults and 51% of those with a chronic illness.¹ Now, it extends into ongoing relational communication undertaken on social networking (SN) websites, recently reported by 47% of online adults in the USA.² Driven by steep uptake of general purpose social media, online health SN is emerging as a vibrant communication technology through which a distributed population of persons with a shared health problem or interest exchanges information and support through personal or community-broadcast channels.³ Engagement may offset the time and attention constraints typical of clinical encounters and provide important avenues for health communication, factors related to improved outcomes.^{4 5}

Background

Little is known about the quality and safety of health-related SN sites.⁶ Despite recognition that the internet in general and social media specifically are viewed by consumers as crucial resources for consumer health information,⁷ data are lacking on the value or impacts of these resources. No formal assessments have been done of alignment of site-published content with medical science and/or clinical practice recommendations (CPRs), accuracy of site- or member-published information, transparency of advertising content, and effectiveness of protections against misinformation and deceptive advertising. Recent studies of member posts and communications within the Facebook community suggest strong demand for emotional support and unsolicited exposure to promotional information including from advertisers.⁸ These risks are traditionally protected against by professional peer review and publication of formally sourced findings. Resources may not be available to support these checks within health-focused SN sites, and sites may eschew these controls in favor of organic peer exchange.⁹ A related concern is that health-focused SNs cater to vulnerable populations who may poorly understand or discount privacy risks under conditions of countervailing need for information and support. Thus, the accessibility and readability of site privacy policies and practices for centralized sharing of member data and opportunities for members to control data are vital. Poor practice in these areas may imperil patients and foreclose opportunities to develop SN for health.

To foster informed decision-making by patients and the clinicians advising them about SN engagement, we sought to characterize the quality of online health-related SN. We focused on sites organized around diabetes, a health problem of global significance,¹⁰ which imposes on patients heavy daily demands for monitoring and management that could be offset by SN.¹¹

Methods

We undertook an observational study of the quality of diabetes-oriented SN websites. We defined SN sites as internet-based social systems that allow a distributed community of individuals to connect, communicate, and share information, and establish a stable personal identity or profile by which individuals can be recognized and connect to other individuals, groups, or collectives. We used a strict approach of reviewing and categorizing site contents using internet published information without requesting additional information or clarification from site administrators. This approach was deemed to result in review of information that would be most reflective of information generally available to users in a fashion that was least invasive to the community and members. Because the study design involved site masking with anonymity of data sources reported at the system level, the IRB approved it and exempted it from full review.

Site selection

SN sites were identified through the Google search engine using ‘diabetes’ as a disease term and ‘social networking,’ ‘community,’ ‘virtual community,’ and ‘forum’ as network function terms. Site inclusion criteria in addition to a keyword match were: site falls within top 20 sites identified through the Google search; English language; no prohibitions on adult access; members can define a unique personal profile or self-description that persists over time; members can interact with other members directly. Sites that focused on multiple illness/health concerns (termed ‘pan health’ sites) were excluded. The review period was October 2008 to July 2010.

Review activities and analyses

Two trained research assistants reviewed sites using a structured protocol covering 28 quality indicators in four domains:

• Alignment of content with diabetes science and clinical practice recommendations (CPRs): these indicators were included to characterize the extent to which site content was complementary to content of traditional authoritative sources and as a means of assessing the potential for synergy and reinforcement of information across sources.

• Safety practices for auditing content, supporting transparency and moderation: these indicators were included to characterize strategies used by sites to monitor content for accuracy, misinformation and potential conflict of interest, recognizing that monitoring could take the form of external and/or internal review of centralized, member (consumer) authored, and commercial (advertising) information.

• Accessibility of privacy policies and the communication and control of privacy risks: these indicators were included to ascertain the extent to which sites prioritized effective understanding of privacy risks among a broad consumer audience and informed consumers about site use of common safeguards for protecting personal information.

• Centralized sharing of member data and member control over sharing: these indicators were included to foster comparison of site practices for sharing information and approaches taken by sites to providing members with opportunities to personally control sharing in the context of social network engagement.

Domain and measure specification served a goal of balancing quality indicators that reflect diabetes science with indicators that reflect the design choices, strategies, and opportunities within the SN medium. Indicators were selected that could be reviewed across sites, dichotomously coded and summed in thematic indices to facilitate within and across site review. Domains and indicators were specified prior to site observation activities, and operational definitions of indicators were developed through team consultation and review of published literature. Wherever possible, indicators were chosen that allowed the team to assess site content relative to standard metrics and measures (eg, definitions of A1c, or CPRs) and internet practice (eg, tools for protecting personal information). To minimize bias from secular changes in diabetes science or temporal variability in site content, thematic areas were reviewed near simultaneously across sites.

The first domain reviewed pertained to alignment of centralized site content with diabetes science and CPRs.¹² We searched site areas designated for persons ‘new to diabetes’ or ‘newly diagnosed’ plus all centrally published content using each site's search function for presence of nine indicators: a medical disclaimer or text presented alongside editorial content that advised site users to consult with a healthcare professional about their care; clear central and correct definitions of glycosylated hemoglobin (A1c) and prediabetes; and text referring to each of six CPRs for diabetes care (routine checkups, annual eye exams, biannual A1c, yearly flu vaccine, annual lipid profile, and smoking-cessation recommendation).

The second domain reviewed pertained to safety practices for auditing site content and supporting transparency of advertising. Use of external resources to audit content was assessed (eg, the Health On the Net Foundation (HON) service,¹³ which charges sites for certification and checks sourcing but not accuracy of published content) as was presence on the home page of a link to an accredited or professional association at which users might reasonably expect to see vetted information. We assessed internal safety practices and resources considering support for formal or informal moderation of user-generated content and its effectiveness for addressing misinformation. We characterized the presence of moderation, clarity of the number of moderators and their credentials, and use of credentialed moderators. Moderation effectiveness was assessed by searching for the presence of member-published content related to availability of a ‘cure’ for diabetes and formal or informal moderation through removal or rebuttal. Once a week for three consecutive weeks, research assistants searched for the term ‘cure’ in the discussion section of each site and surveyed the presence and persistence of content promoting a cure for, or reversal to, diabetes within the five most recent discussions. Sites with member-posted content that did not have or removed/rebutted misinformation about a cure within the 3-week window were deemed ‘free’ of such content, while sites hosting discussions in which member-posted misinformation describing a diabetes cure was found that persisted (ie, was not removed or rebutted) were deemed ‘not free.’ Member-published content that referred to a wish for or anticipation of a cure or a similar out-of-context use of the term was not counted as misinformation. Given the potential for advertising activities to influence perception of disease risk or treatment choice,^{14 15} we assessed the presence of pharmaceutical advertising on the site home page, the transparency of advertising content as distinct from editorial content, and the safety of advertising content. The latter item was assessed by searching for the presence of misleading or deceptive advertising promoting a diabetes ‘cure’ or ‘reversal’ to diabetes using the above 3-week schedule to allow a review of time-varying safety risks.

The third domain reviewed pertained to privacy policies and the communication and control of privacy risk. Sites were surveyed for the presence of a privacy policy and its location within two clicks of the home page. Also, we checked for whether a common technical practice of placing on a member's computer a software code called a ‘cookie’ which has privacy ramifications was clearly defined, the readability of the policy based on a quantitative estimate of the number of school years (grades) required to understand the published policy. We estimated actual grade-level readability of a sampled portion of each site's published privacy policy using the Gunning Fog readability index, a standardized formula that defines the number of years of education completed required to understand a text¹⁶ and dichotomized the index at grade 8, the threshold for near-universal readability. Finally, we assessed use of any of three controls for protecting privacy of individual level data provided by members, based on a review of the privacy policy and/or privacy practices description and considering as protective use of encrypted media for data storage, and/or a secure socket layer to transmit data, and/or an external agent to audit security practices of the site.

The fourth domain reviewed pertained to site central practices for sharing member data and member control of sharing and encompassed four indicators. Sites were reviewed for data elements required for membership to ascertain which if any contact, demographic, and geographic locators were on file with the site in addition to any individually posted/published data. These descriptive measures were not included in the index of privacy risks. Next, site policies and terms of use were reviewed to determine practices for sharing members' data, considering sharing practices for aggregate deidentified information, individually identified information, and also ability of members to exert control over sharing of their posted data including through options to restrict access to individually posted data to community members and to restrict the viewing of a member's personal profile page to selected friends or affiliates.

All measures were coded dichotomously to facilitate within- and across-site review such that evidence of medical alignment, use of various mechanisms to ascertain safety and transparency of content, availability and readability of privacy policies, use of measures to protect disclosure of identified data, avoidance of centralized sharing of identified data, and options for member control over sharing were positively scored (coded as ‘1’), and the absence of these measures or counter practices was neutrally scored (coded as ‘0’). In this fashion, higher index values connote better practice. Findings are reported in thematically organized tables with by- and across-site values given for discrete indicators and thematic indices. A by-site comparison of quality considering all four thematic indices is reported as a summary graph to identify best-practice patterns.

Results

General characteristics of diabetes-related online SN sites

Eleven sites met the study inclusion criteria. One closed during the review period and was excluded from the report for incomplete information. Sites have founding dates between 2001 and 2008. All address multiple diabetes subtypes (ie, Type 1, Type 1.5, Type 2, prediabetes, and gestational diabetes). All target adults, and one focuses on youth ages 13 and older and their adult care givers. The average community membership was n=6707. Member activity varied within a series of randomly monitored 24 h periods: four sites published over 100 new messages per 24 h period, while two sites were minimally active and published fewer than five new messages in that time. The level of site activity and membership size were not correlated. Seven sites were for profit. Three sites belonged to a professional association or organization. Across sites, several types of promotions were available to members, including coupons for diabetes supplies (four sites), information about clinical trials (five sites), and information about diabetes-related events (seven sites).

Alignment of site content with diabetes science and clinical practice recommendations

Across sites, there was a bimodal pattern of alignment of site content with diabetes science and CPRs, reported in table 1. Five sites had alignment indices with values in the top third of the nine-point scale, and three sites had values in the bottom third of the scale. Where a mixed pattern was evident (sites G, H, I), no clear pattern was present. Across sites, the lowest areas of alignment were for presence of a medical disclaimer or reminder to discuss care with a provider which was present on three sites; and publication on site of a definition of A1c, present on five sites, of which none defined A1c levels according to standards for effective disease management. Four sites provided recommendations for all six CPRs assessed; three sites had no information on these CPRs.

Table 1.

Alignment of centralized content with diabetes science and clinical practice recommendations

Medical alignment	Social-networking site
	A	B	C	D	E	F	G	H	I	J
Recommendation or ‘disclaimer’ to discuss regimen with healthcare provider	Yes	No	No	Yes	No	No	No	No	No	Yes
Clear central definitions, consistent with care standards
1. HbA1c	No	Yes	No	Yes	No	Yes	Yes	Yes	No	No
2. Prediabetes	Yes	Yes	Yes	Yes	No	Yes	Yes	Yes	Yes	No
Clear central information on clinical-practice recommendations
1. Routine checkups	Yes	Yes	No	Yes	No	Yes	No	Yes	Yes	No
2. Annual eye exams	Yes	Yes	No	Yes	No	Yes	Yes	Yes	Yes	No
3. Biannual HbA1c	Yes	Yes	No	Yes	No	Yes	No	Yes	Yes	No
4. Yearly flu vaccination	Yes	Yes	No	Yes	No	Yes	Yes	Yes	No	No
5. Annual lipid profile	Yes	Yes	No	Yes	No	Yes	Yes	No	Yes	No
6. Smoking cessation	Yes	Yes	No	Yes	No	Yes	Yes	Yes	Yes	No
Alignment index (sum above)	8	8	1	9	0	8	6	7	6	1

Scored as yes=1; no=0.

Safety of diabetes SN sites

A bimodal pattern was observed for safety practices (table 2); three sites' safety index values fell in the top third of the 10-point scale, and the remainder fell in or near the bottom third. With respect to external checks on the quality of published editorial content, two sites indicated that they subscribe to an outside agency that audits site content by ensuring content is sourced; however, the accuracy of published information was not verified under that service. Five sites publish a link to an outside diabetes organization on the home page, which may facilitate members' access to formally vetted diabetes information or guidelines. Internal checks on user-generated or member content through lay- or professional moderation were suggested in seven sites. Specific information about numbers or credentials of moderators was available for three of these sites. One site indicated use of professionally accredited moderation (ie, moderator has professional grounding in diabetes clinical science, nutrition, nursing, other). Member-posted misinformation about a cure for diabetes was found on six sites; four of these were moderated sites. Four sites were free of member-posted misinformation about a cure, of which three were moderated sites. Nine sites allowed advertising. One site had pharmaceutical advertising on its home page. Four sites clearly distinguished advertising from editorial content, and six sites were free of deceptive advertising about a ‘cure.’ Sites A, D, and H were identified as better-practice sites; they had transparency of content and no pharmaceutical advertising on their home page or deceptive advertising. Two sites lacked a clear distinction between advertising and editorial content and had advertisements promoting a ‘cure’ for diabetes.

Table 2.

Safety practices for auditing content and supporting transparency and moderation of content

Quality of information	Social-networking site
	A	B	C	D	E	F	G	H	I	J
External checks on editorial content
Evidence of outside audit of site content	No	No	No	Yes	No	No	No	Yes	No	No
Link to diabetes association on home page	Yes	No	No	Yes	No	Yes	Yes	Yes	No	No
Internal checks on user-generated content
Some form of forum moderation present	Yes	Yes	No	Yes	Yes	Yes	Yes	Yes	No	No
No of moderators available clearly marked	Yes	Yes		Yes	No	No	No	Yes
Credentials of moderators clearly stated	Yes	Yes		Yes	No	No	No	No
Evidence of professionally accredited moderators	No	No		Yes	No	No	No	No
Free of member-posted misinformation about a ‘cure’	Yes	No	Yes	No	No	Yes	No	Yes	No	No
Advertising content
Free of pharmaceutical advertising on home page	Yes	Yes	No	Yes	Yes		Yes	Yes	Yes	Yes
Clear distinction between editorial and advertising content	Yes	No	No	Yes	No		No	Yes	No	Yes
Free of deceptive advertising of ‘cure’ on site	Yes	No	Yes	Yes	No		Yes	Yes	Yes	No
Safety index (sum above)	8	4	2	9	2	3	4	8	2	2

Scored as yes=1; no=0. Site F did not contain any advertisements.

Characteristics of privacy policies governing diabetes SN sites

Privacy policies were published on eight of 10 sites, and the majority of these made them available within two clicks of the home page so they could be readily found (table 3). All sites use ‘cookies’ to track user preferences, which can risk privacy. Only four sites both mentioned and defined cookies in their privacy policy (two mentioned but did not define cookies). None of the sites had policies that were readable, based on a standardized readability assessment dichotomized at or below the eighth grade. The actual readability of sampled text across sites ranged from a low of 9.5 years of education (greater than first year high school in the USA) to a high of 18.4 years (6 years beyond high school completion in the USA). The rigor of privacy practices was low, considering site use of any of three best practices for safeguarding privacy—data storage using encrypted media, data transmission through a secure socket layer, and use of an external agent to audit security practices. No sites met all three criteria, one site (F) reported two of the three, and the remainder did not use any of the three approaches, or practice could not be determined based on published information. Thus, for the majority of sites, information about how the site addresses privacy was available but inaccessible based on reading level with little evidence that practices were backed up by technical strategies.

Table 3.

Accessibility of privacy policies and the communication and control of privacy risks

Published privacy policy	Social-networking site
	A	B	C	D	E	F	G	H	I	J
Privacy policy available	Yes	Yes	Yes	Yes	No	Yes	No	Yes	Yes	Yes
Policy within two clicks of main page	Yes	Yes	No	Yes		Yes		Yes	Yes	No
Technical term defined	Yes	Yes	No	Yes		Yes		No	No	No
Readability of policy at or below eighth-grade level	No	No	No	No		No		No	No	No
Any of three security measures to protect information	No	No	No	No		Yes		No	No	No
Privacy Policy Index (sum above)	3	3	1	3	0	4	0	2	2	1

Scored as yes=1; no=0.

Practices for protecting privacy of member data

Multiple categories of information were required from potential members to register at most sites, although the accuracy and veracity of information are not checked or confirmed (table 4). All sites required that members provide contact information; three sites required demographic information; two sites required geographic information. One site does not share aggregate deidentified information, and six sites do not share personally identifying information. Variation in availability of member controls around information sharing was found across sites. None of the sites support a mechanism by which users can restrict access or sharing to community members. At three sites' users can restrict profile page access to their friends and associates. One site required members to provide multiple categories of information for membership, shared individually identified data, and offered no member controls for use or visibility of profile data. These practices were published in that site's privacy policy; however the readability level of that policy was at the high end of observed values (approximately 16 years of school which equals 4 years of education after high school in the USA).

Table 4.

Centralized sharing of member data and member control over sharing

Data required for membership (not counted in index)	Social-networking site
	A	B	C	D	E	F	G	H	I	J
Contact: name, email	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes	Yes
Demograhic: birth date, age, gender, occupation	Yes	No	No	No	No	Yes	No	No	Yes	No
Geographic: country, state/region, zip code	No	No	No	No	No	Yes	No	No	Yes	No
Data shared by sites
Site does not share aggregate/personal deidentified data	No	No	No	No	NA	No	NA	Yes	No	NA
Site does not share personally identifiable data	Yes	Yes	Yes	Yes	NA	No	NA	Yes	NA	Yes
User control over data
Access to community posted data is restricted to members	No	No	No	No	No	No	No	No	No	No
Members can restrict profile view to private or friends only	Yes	No	No	No	No	No	Yes	No	No	Yes
Privacy practices index (sum above)	2	1	1	1	0	0	1	1	0	2

Scored as yes=1; no=0. NA, not available/cannot determine=0.

Consistency of better practice

Across-site analysis of each of the four quality domains showed a clustering of better practice at three sites (A, D, and H) and is reported in figure 1. Within the observed range of site quality measures, sites that have high levels of alignment with diabetes science and clinical practice recommendations also have high levels of content safety, transparency, and protection from misinformation. None of the sites was completely aligned with diabetes science or published information on all six CPRs assessed, few used standard technical tools to protect personal information, and communication of risk and transparency of advertising were weak.

Figure 1.

Composite quality and safety index.

Discussion

In this first study of the quality of online SN for health, we characterized the 10 most frequently referenced diabetes sites along 28 quality indicators pertaining to four practice domains. We documented variation in practice by domain and across sites to characterize better practice. Variability was evident, as was the ability of a minority of sites to offer greater quality and safety protection. With respect to medical alignment, safety, and transparency, notable gaps even among better practice sites were around publication of a medical disclaimer or recommendation reminding users to discuss their care with a provider, and provision of a link to a specific disease or professional association on their home page. Gaps may be oversights or intentional omissions. Lay-led and community-based networks may be choosing to amplify the community voice above the professional citation or resource. Alternatively, professional associations and the medical infrastructure may be ignoring these communities. Another notable gap area was around moderation, found among half of the sites. Information about moderation was scarce, and the presence of misleading content about a cure for diabetes raises questions about its effectiveness. Advertising content and pharmaceutical industry presence were observed on nine sites, although eight of these did not have pharmaceutical advertising on their home pages. Transparency around advertising content was missing on five sites, a possible red flag for misinformation, commercial influence, and even conflict of interest.¹⁷ Indeed, three sites contained advertisements promoting a ‘cure’ for diabetes.

Publication and easy identification of privacy policies was near universal among sites, but availability may be undermined by policies' readability levels, which precluded broad accessibility. In the context of poor communication of privacy risks (ie, through a readable policy and clear definition of common risks to privacy), only three sites afforded members opportunity to control access to their profile information, and only one site reported using any of three technical strategies to protect privacy in an assessment of industry-level approaches to safeguarding information. The prevailing assumption governing privacy within sites appears to be ‘proceed at your own risk,’ where ‘terms of use’ substitutes for informed consent.

Recommendations

The voluntary nature of SN, its disconnection from institutional authorities, and its alignment with patient/consumer-driven models and philosophies may seem to reduce our obligation for protective action to improve quality and support safety. Moreover, the diffusion of SN memberships over countries and cultures obscures locus of responsibility for doing so. Nevertheless, actionable steps for improving quality and protecting populations made vulnerable by illness are warranted and include:

• ensure easy identification of where to get help, guidelines for care, and professionally recognized content within the larger set of broadcast information;

• post information about the use of credentialed moderators and their number so users can be informed about protection;

• enlist periodic external review of samples of member discussions to protect from misinformation and support effective moderation;

• clearly and consistently flag commercial content and commercial members;

• ensure that privacy policies are easy to find, clearly readable by the majority of healthcare consumers;

• provide for member control over sharing of personal health information within and beyond the network;

• use industry standard approaches to protecting individual health information and sharing defaults that emphasize protection of personal health information.

An effective system of site quality review is needed that encompasses the study domains and extends to periodic safety monitoring of published content given that this information is time-varying. Patient, patient advocacy, and medical communities should give consideration to the type of mechanisms that may be available to support cost-efficient peer or professional review. To be effective, a quality-improvement mechanism will need to balance values of individual autonomy and community ownership with conditions for innovation and consumer protection. Traditional institutional and regulatory approaches to overseeing patient safety and privacy poorly fit the online SN medium, and there is no clear authority or agency empowered to protect patients within this medium. Voluntary review by members of professional associations is one option that might be pursued as a service requirement for association membership. Another option is a peer-based review system in which members and leaders of a collection of SN communities ‘site visit’ each other to offer support and guidance around quality criteria. Sites that support these efforts and meet criteria might be flagged with a badge indicating participation in a quality-monitoring system. Such a system might inform users, contribute toward quality, and improve use of SN to ameliorate the effects of diabetes, a debilitating chronic illness with an expanding global footprint.

Strengths and limitations

This study adds to the growing literature on health-related SN by characterizing quality and safety issues at the system level. This is a new area with limitations. We report on observed practice and did not contact site administrators to obtain unpublished practice information. This allowed us to characterize apparent risks and protections as a user might encounter them in weighing possible benefits and harms. Actual protections and practices may differ and could be more rigorous if sites under-reported protective tools. We reviewed the most frequently referenced English language sites centered on diabetes using specific keywords. The full universe of diabetes sites or other health sites is not represented. We identified sites using the Google search engine, a site with broad consumer recognition and reach. Use of alternative search engines could result in identification of additional sites that were not included in this study, especially in light of the potential that websites could promote the visibility of their site on the Google platform through financial arrangements with Google. We used multiple measures to indicate practice along different dimensions, drawing on recognized standards where possible. This should protect against threats to the reliability and validity of findings; however, no gold standard for measuring the quality of SN practice for health yet exists.