The quality of informational social support in online health communities: A content analysis of cancer-related discussions

Abstract

Objective

Informational social support is one of the main reasons for patients to visit online health communities (OHCs). Calls have been made to investigate the objective quality of such support in the light of a worrying number of inaccurate online health-related information. The main aim of this study is to conceptualize the Quality of Informational Social Support (QISS) and develop and test a measure of QISS for content analysis. A further aim is to investigate the level of QISS in cancer-related messages in the largest OHC in Slovenia and examine the differences among various types of discussion forums, namely, online consultation forums, online support group forums, and socializing forums.

Methods

A multidimensional measurement instrument was developed, which included 20 items in a coding scheme for a content analysis of cancer-related messages. On a set of almost three million posts published between 2015 and 2019, a machine-learning algorithm was used to detect cancer-related discussions in the OHC. We then identified the messages providing informational social support, and through quantitative content analysis, three experts coded a random sample of 403 cancer-related messages for the QISS.

Results

The results demonstrate a good level of interrater reliability and agreement for a QISS scale with six dimensions, each demonstrating good internal consistency. The results reveal large differences among the social support, socializing, and consultation forums, with the latter recording significantly higher quality in terms of accuracy (M = 4.48, P < .001), trustworthiness (M = 4.65, P < .001), relevance (M = 3.59, P < .001), and justification (M = 3.81, P = .05) in messages providing informational social support regarding cancer-related issues.

Conclusions

This study provides the research field with a valid tool to further investigate the factors and consequences of varying quality of information exchanged in supportive communication. From a practical perspective, OHCs should dedicate more resources and develop mechanisms for the professional moderation of health-related topics in socializing forums and thereby suppress the publication and dissemination of low-quality information among OHC users and visitors.

Introduction

Informational social support as a main resource in OHCs

Online health communities (OHCs) have undoubtedly become a crucial part of public health. OHCs and their use have an important impact on the health-related outcomes of patients, caregivers, and Internet users in general^1–3 and the dynamics of their interactions with health professionals.^4,5 OHCs facilitate the formation of new connections among users as well as the exchange of different forms of online social support aimed at meeting various health-related needs.^2,6,7 Online social support refers to the social resources produced through online interpersonal communication, the purpose of which is to provide assistance to those who need of it.⁸ In OHCs, there are various forms of online social support exchanged among users^8,9: emotional support, which comprises expressions of empathy, understanding, affection, acceptance, and care when dealing with difficult health situations; network support, which consists of connecting with others and broadening social networks; instrumental support, which refers to the provision of material and/or financial goods and services; and informational support, which includes the provision of advice, useful information, guidance, and suggestions for coping with health issues or the management of health conditions. Among these, informational social support has been deemed one of the most frequently exchanged forms of social support among OHC users and one of the main reasons for OHC usage.^2,6,10 This is not surprising, since informational social support has been shown to be beneficial for OHC users as it improves their general health status¹⁰ and equips them with the knowledge and skills to reach their health-related goals.¹¹

Moreover, informational social support helps users compensate for the possible lack of support from health professionals¹² and leads to individual and collective empowerment¹³ as well as empowerment in the relationship with the physician.⁵ Online exchange of informational social support in OHCs can be an important component of digital stigma coping strategies and a resource especially for individuals experiencing visible or physical stigma due to conditions such as obesity, hair loss, and physical disabilities.¹⁴ Furthermore, informational social support received through OHCs is particularly beneficial for patients with chronic health conditions, such as cancer, who appreciate receiving relevant information on possible treatments, disease recurrence, quality of care, and the experiences of others with similar conditions.^15–18 Despite convincing empirical support for the beneficial effects of informational social support on OHC users, to the best of our knowledge, no study has comprehensively investigated the quality of information exchanged in supportive communication within OHCs. OHCs are not immune to issues of misinformation, disinformation, inaccurate information, and low-quality information in general.¹⁹ We believe that this is an important shortcoming that needs to be addressed through an investigation of the Quality of Informational Social Support (QISS) in OHCs.

Challenges relating to the quality of information exchanged in supportive communication in OHCs

The main rationale of our study centers on the finding that the quality of information exchanged in supportive communication among OHC users is often assumed and not actually examined. This has often been mentioned as one of the main limitations of research on informational social support exchanged in online support groups and OHCs.² Although earlier studies saw this as a minor problem—for example, Esquivel et al.¹⁵ found that 0.2% of the statements in a cancer-oriented OHC were false and misleading—at least one recent study has suggested that approximately half of the information posted in advice in OHCs is generally appropriate.²⁰ Moreover, recent studies have shown a high prevalence of health-related misinformation on social media.^19,21 This is not surprising as, similar to other social media, gatekeeping in OHCs is weak or nonexistent, and anyone can post messages that are visible to other users. Some OHCs embody self-corrective strategies¹⁵ and involve health professional moderators.^22,23 These strategies can, to some extent, limit the spread of low-quality information. However, OHC managers are often limited in committing resources to employ suitably qualified health professional moderators.²⁴ Besides the online consultations provided by health professional moderators, OHCs also include spaces for general everyday discussion, such as socializing forums. These are usually less regulated but still represent a space for the exchange of health-related information and are usually not verified or evaluated before they are publicly published in OHCs.

Research on the quality of online health information demonstrates worrying findings. The quality of websites on rare diseases is rather low,²⁵ and the most frequently visited websites on gynecological cancer are of variable quality.²⁶ The quality of online information regarding cervical cancer is moderate, with marginal amounts of scientific references and information on side effects.²⁷ Furthermore, YouTube videos on cesarean delivery methods received a score of 0 in the scientific evidence category, with almost no links to scientific references.²⁸ An analysis of 200 websites on breast cancer treatment options revealed that among commercial, nonprofit, and professional websites, fewer than 50% received a high-quality score according to criteria from the Journal of American Medical Association.²⁹

The problem of low-quality information is magnified when OHC users are not sufficiently equipped with e-health literacy to detect this low quality and are, thus, susceptible to misleading information. Research reports that internet users in general are not enough aware of the problem of quality of online health information.³⁰ Sun et al.³¹ demonstrate that users of online health information are overly reliant on subjective criteria in judging the quality of information received in OHCs. Even more worrying, individuals with low e-health literacy rely on criteria that are not commonly cited in quality guidelines, such as the quality of pictures, position in search engines, or celebrity endorsement.³² The problem is further amplified in light of the findings that OHCs can be characterized by discrepancies between what users think is credible information and what is credible from a scientific/professional perspective. For example, in a study by Keselman et al.,³² there was no correlation between the credibility of the source and the perceived credibility of the information provided.

What are the outcomes of receiving informational social support based on low-quality information? There is no direct research on this topic, but it has been suggested that inaccurate advice shared online can lead to misguided therapies, with harmful consequences for patients.^6,33,34 With cancer patients, in particular, the congruence of information with medical expertise is crucial; conversely, misleading informational support without evidence can be dangerous for the patient,³⁵ which can lead them to discontinue certain cancer therapies.³⁶ Before researchers can investigate the effects of the QISS, we need to conceptualize this phenomenon and develop a scale to measure it.

Conceptualizing the QISS

The question of the quality of online health-related information has been the subject of extensive research since the immediate aftermath of the emergence of the World Wide Web. However, this research has mostly focused on various websites with professionally published articles and not on interactive platforms, such as OHCs, in which information is constantly produced in interactions among users.³⁷ Investigations into the quality of information in OHCs have been limited to the perceived credibility of information, which is an important phenomenon, however, as mentioned above, it does not necessarily correlate with the objective quality of information. To our knowledge, no study has comprehensively examined the quality of the information exchanged in supportive communication among OHC users.

In this study, we propose a conceptualization of the QISS. To do this, we apply criteria relating to the quality of online health information that are of relevance to the substantive and formal aspects of information published on websites^38,39 and exchanged in informational social support. Since informational social support pertains mainly to information exchanged in communication among OHC users,^2,40 the QISS pertains to the accuracy and completeness of this information against the guidelines and findings of medical science, the readability of the informational support provided, and the trustworthiness of the author of the messages providing informational support. The QISS concept is not interested in the presentational and structural characteristics of websites, such as the quality of the table of contents, the efficiency of internal search engines, or the visual presentation of information. It focuses only on the content of the messages exchanged in supportive communication among OHC users, that is, publicly visible messages.

In addition to the application of relevant dimensions of the quality of online health information (on websites), we aimed to consider the assumption that informational social support occurs in the process of interpersonal communication among users of OHCs.⁴⁰ Consequently, we believe that some aspects of the (quality) of online interpersonal communication need to be considered in the conceptualization of the QISS. We can find these aspects in the literature on the quality of online conversations and online deliberation,^41,42 where we extract the dimension of justification—defined as the degree to which arguments are provided while exchanging information in supportive communication in OHCs. Recent research has also suggested that an important aspect of the QISS is to provide information in OHCs that is supported with material that increases its plausibility, such as scientific evidence.^32,36 Drawing on the dimensions of the quality of online health information and online deliberation, we can, therefore, summarize the following relevant dimensions of the QISS concept:

1. Accuracy is defined as the degree of congruence of the health information provided with generally accepted professional standards and practices.³⁸ More specifically, it is about the extent to which health-related advice, knowledge, or guidelines are scientifically accurate and consistent with valid medical guidelines.⁴³

2. Completeness pertains to the comprehensiveness, balance, and extent of health-related advice and knowledge.⁴⁴ In other words, it describes the extent to which the health-related message in OHC represents all relevant, positive and negative aspects of the topic under discussion.

3. Readability is defined as the extent to which the content of messages providing informational social support is clear and understandable to readers without a medical background or training.³⁹ The informational social support on health-related topics is readable when it is delivered at a level that can be understood by readers without any medical training and education.

4. Trustworthiness pertains to the impartiality and neutrality of the author and the absence of misleading information. In other words, a message is considered trustworthy if the content is independent of commercial interests and the health-related information is given in an unbiased manner.³⁹

5. Justification is defined as the degree to which valid reasons are provided for messages containing informational social support.^45,46 High level of justification pertains to messages that provide convincing scientific and/or medical arguments for health-related statements in the message.

Aims of the study and research questions

The main aims of this paper are threefold. The first is to develop a measure of the QISS in OHCs and empirically test it in the context of Slovenia's largest OHC. By introducing the concept and the measure of the QISS, we provide the research with a tool that can help answer the pressing question of what impact the sharing of problematic quality information in online supportive communication has on individuals’ health and their relationship with their physician. Second, by using quantitative content analysis, we aim to investigate the QISS level of this particular OHC in cancer-related discussions. Cancer is one of the most common chronic conditions in Slovenia⁴⁷ and since OHCs are very important to cancer patients,⁴⁸ the QISS in relation to these discussions is a crucial area of study. The third aim is to investigate QISS differences among different OHC discussion spaces as it has been established that contemporary OHCs are often comprised of various types of services, including online supportive communication between peers, online consultations with health professionals, spaces intended for socializing and for users to connect with others, and even clinical trial access.⁴⁹ Similarly, Slovenia's largest OHC, Med.Over.Net (MON), comprises three types of discussion forums: online consultations with health professionals, online support groups, and socializing forums. Online consultation forums are typically structured into Q&A sections, where users/patients ask questions and health professionals (moderators) provide answers. These types of discussion spaces are usually strictly moderated. Online support groups are intended for the exchange of advice and experiences among patients with similar symptoms and diseases and are usually somewhat moderated, though mostly in terms of interactive moderation, which is limited to facilitating and managing discussions and not correcting content.⁵⁰ Lastly, socializing forums provide spaces for users to converse about daily matters, from health to politics, culture, and trivia. These kinds of discussion spaces are important for the sustainability of the online (health) community⁴² and are typically lightly moderated or not at all. Research^23,24 indicates that the information provided by experts in online consultations is the most credible. However, at least one study found no differences in the quality of online information between support groups and medical institutions.²⁵ Consequently, there is a need to investigate the potential differences and assesses the QISS provided in different discussion spaces in OHCs. Accordingly, our main research question is as follows: What is the QISS in the largest OHC in Slovenia regarding cancer-related discussions, and to what extent does this quality differ among online support groups, socializing forums, and online consultations?

Method

Description of the OHC under investigation

Our study was performed in collaboration with Med.Over.Net (MON), Slovenia's largest and most visited OHC and among the top 15 most visited websites in Slovenia.⁵¹ MON was founded in 2000 and covers areas such as health, medicine, social work, law, and education. In 2020, it had over 400,000 monthly visits and on average of 70,000 monthly users. It is structured similarly to other OHCs comprising sets of discussion forums within which there are a number of forum threads and, within them, messages (between 1 and 1000 messages). Furthermore, MON's forums reflect the above typology of discussion spaces: online consultations with health professionals, online support groups, and socializing forums. In online consultations, health professionals from different health-related areas serve as moderators, answer users’ questions, and decide on the degree to which discussions among the forum's users are allowed and encouraged. The engagement of health professional moderators in MON is voluntary, that is, there is no financial compensation. Online support groups are intended for the exchange of information among peers and include moderators, who facilitate discussions, and usually patients, ex-patients, or representatives of patient associations. Socializing forums seem to be the most vibrant part of MON and are very rarely moderated, if at all. In May 2020, MON had 128 online discussion forums (70 online consultations with around 90 health professional moderators, 22 online support groups, and 36 socializing forums), around 1.5 million forum threads, almost 12 million published forum posts, and around 120,000 registered users.

Some forums within MON are explicitly dedicated to the topic of cancer (e.g. “How to live with cancer?” “Lymphoma and leukemia,” and “Legal aid for cancer patients”). A number of forums are not explicitly intended for cancer-related discussions, yet such discussions also appear in other forums (e.g. “Family medicine” and “Food safety”). Our aim was to analyze cancer-related messages in all forums in which they were published.

Dataset

MON's managers gave us access to the dataset of all messages published from 2015 to 2020. The dataset consisted of 2,982,564 messages published in 280,357 forum threads within 128 forums. Table 1 shows that within the socializing forums, there was the largest number of forum threads (n = 253,455) and that these threads contained an average of 15.5 messages. Conversely, the online consultation forums had, on average, the lowest number of messages per thread (5.37), while the online support groups had the lowest number of forum threads (n = 15,194) and an average of 13.71 messages per thread.

Table 1.

Number of messages and threads per forum type.

Type of forum	N of threads	Mean (messages)	SD (messages)	Min	Max
Online consultation forums	52,425	5.37	25.88	1	1000
Online support group forums	15,194	13.71	56.16	1	1000
Socializing forums	253,255	15.50	33.72	1	1000

In order to procure a final dataset of messages providing informational social support on cancer-related topics, we followed four steps: (1) machine learning procedures to identify cancer-related messages in the original dataset; (2) sampling of forum threads with cancer-related messages; (3) manual identification of cancer-related messages eliciting informational social support; and (4) quantitative content analysis and expert coding of the QISS. These steps are described in greater detail in the following subsections, while Table 2 summarizes the resulting number of forums, threads, and messages at each step.

Table 2.

Procedure for obtaining a random sample of cancer-related messages.

Steps in the procedure	Number of forums	Number of forum threads	Number of messages	Number of cancer-related messages
Total	128	275,268	2,979,905	17,701
Forum threads with at least one cancer-related message	103	7340	204,011	17,701
Forum threads with at least two messages	101	6794	203,465	17,155
Forum threads with 100 messages or less	101	6367	118,137	14,941
Forums with at least five forum threads	60	6288	117,527	14,810
A sample of randomly selected forum threads and messages	60	696	6882	1079
Messages with information support about cancer	51	303	403	403

Identification and sampling of cancer-related messages

In order to identify the cancer-related messages in the original dataset, we conducted a supervised machine-learning procedure. We examined the efficiency of various machine learning algorithms on a learning set containing 1287 messages, of which 468 (36%) were about cancer. The messages were first randomly selected from the database of messages on MON, then manually based on a list of 54 predefined cancer-related keywords (a step performed to increase the proportion of cancer-related messages).

These messages were used as the learning set for a random forest algorithm, a naive Bayes classifier, and a support vector machine, which are commonly used classification algorithms.⁵² The process of text preprocessing included cleaning (i.e. removing special characters such as HTML tags), lemmatization, tokenization, and stop word removal (words from the list of stop words and words with frequencies of less than five were removed). Regarding lemmatization, we used Obeliks, a lemmatizer for the Slovene language.⁵³ The performance of the models was evaluated using a 10-fold cross-validation.⁵⁴ The random forest algorithm yielded the best performance (92% sensitivity, 92% specificity, 87% accuracy, 89% F1 score, and a 92% area under the curve) and was, therefore, used to detect cancer-related messages.

Through the random forest algorithm, we identified 17,701 cancer-related messages (from 2,979,905 messages) appearing in 7340 forum threads (see Table 2). To examine informational social support, only forum threads with a minimum of two messages and a maximum of 100 messages were considered. The minimum of two messages per thread was considered a necessary condition for the exchange of informational social support between two actors,⁴⁰ while the maximum of 100 messages was decided as an arbitrary threshold to assume the maximum length of a discussion on the same topic. This maximum was also defined for practical reasons as the coders needed to read the whole forum thread before coding individual messages containing informational social support. In addition, only the forums with at least five forum threads were considered to achieve a representative sample of forum threads within each forum. These restrictions reduced the number of forums to 60, the number of forum threads to 6,288, and the number of cancer-related messages to 14,810.

From each forum, a random sample of five forum threads was drawn. All messages were selected within each randomly selected forum thread unless the number of messages was greater than 10, in which case, 10 messages were randomly selected. A total of 1079 messages on cancer were selected.

Three experts (authors) rated the presence of informational social support in each of the above-identified messages. The procedure was as follows: each of the experts read all threads containing the above-identified cancer-related messages and, for each message, decided whether it provided informational social support (yes or no). To be able to make this decision, the experts were first acquainted with the definition of informational social support and how it is manifested in the forum messages, thereby distinguishing it from other forms of social support. This definition and explanation included various categories, as identified by,² such as advice, referral to experts, situation appraisal, and teaching in terms of providing factual information about various aspects of the disease. The experts also followed the instruction that messages providing informational social support may include advice, guidelines, suggestions (about therapies, symptoms, health system, drugs, lifestyle, and experts), explanations (e.g. giving medical reasons for a certain situation, therapy, etc., providing links to research, statistics, and other sources), or referrals to doctors, health professionals, and institutions. In addition, they were informed that informational social support generally included statements referring to the outside, objective world (claims about therapies, symptoms, and actors).

If the experts were unsure as to whether a message provided informational social support, they were instructed to put an asterisk next to the message ID, which occurred in seven cases. All three experts independently rated all messages. The expert ratings were followed by a meeting, where all inconsistencies were identified, messages with asterisks discussed, and all discrepancies resolved. Consequently, complete agreement was achieved, eliminating the need to calculate statistical parameters for interrater agreement and reliability.

The final sample contained 403 cancer-related messages with informational social support. These messages appeared in 303 forum threads from 51 forums (37 online consultations, 9 online support groups, and 5 socializing forums).

Measure of the QISS and interrater agreement

QISS measure development. In developing a QISS measure for OHCs, we first reviewed the most commonly used health information quality assessment tools, such as DISCERN,⁵⁵ the Health On the Net Foundation's code of conduct,⁵⁶ and the Journal of the American Medical Association benchmark⁵⁷ and adopted a number of items originally developed to measure the quality of health information (published on various websites). Although the items from these tools are not automatically applicable to OHC discussions, it has been claimed that, specifically for the DISCERN tool, it can be used to evaluate any text-based health-related information.³² In addition to the adoption of relevant items, we also developed a pool of new items on the basis of the essential definitions of the quality criteria (i.e. accuracy, completeness, readability, and trustworthiness) by following an established procedure.⁵⁸ The items for the dimension of justification were adopted on the basis of the justification dimension of the deliberative quality index.⁴⁶ For the production and adoption of items for the coding scheme, we followed four steps:

1. Formation of the initial pool of 39 items based on the adoption of items from existing measures of the quality of health information and communication and the development of new items from the definitions of the dimensions;

2. Optimization of a set of 31 items based on expert reviews of the initial pool of items (the authors evaluated the items, and eight items were excluded on the basis of scores);

3. Development of coding scheme with coding instructions;

4. A pilot study and testing of the coding scheme were conducted on a separate sample of cancer-related messages (n = 55) from MON's discussion forums. The research team and three expert coders (doctors) coded a sample of messages and evaluated the content validity of the scale and, thus, the extent to which the items of the QISS scale included in the coding scheme represented and measured the QISS dimensions. The coded data and coding process paradata were descriptively analyzed. The results helped us identify potential problems with the wording or understanding of the items included in the coding scheme and led to the optimization of the coding scheme and the QISS scale for the quantitative content analysis.

All the items in the final coding scheme were formatted as questions or statements, to which the coders provided agreement on a Likert-type scale ranging from 1 to 5. Each message in the sample was independently coded by three coders, who were doctors specialized in the field of oncology from the Institute of Oncology in Ljubljana. All three coders participated in the training, in which they were given instructions on how to code. The coding scheme was developed in the web survey tool 1ka.⁵⁹ All three coders coded all the messages in the sample (n = 403). The coding was conducted between October 2020 and January 2021. The procedure instructed the coders to read the whole forum thread before coding individual messages providing informational social support within that thread. More information about the coding scheme can be found in Multimedia Appendix 1.

The quality of the developed QISS measure was investigated in a two-step process. First, we investigated intercoder agreement and intercoder reliability, and in the second step, we ran a confirmatory factor analysis (CFA) to test the assumed multidimensional structure of the QISS measurement instrument.

Intercoder reliability and agreement. We analyzed both intercoder reliability and intercoder agreement, which are often taken as synonymous, but there are certain important differences between them. Interrater reliability pertains to the degree to which the coders’ responses are equal⁶⁰ and is commonly computed using the intraclass correlation coefficient (ICC)⁶¹ and Kendall's W coefficient. Here, we considered that three coders evaluated each message with each item (two-way model) and that, in the analysis, the average score of all three coders would be taken into account. It was considered that ICC values above 0.5 were satisfactory.⁶¹ We also considered Kendall's W coefficient, which is a type of correlation coefficient on an interval [0, 1] where a value of 1 corresponds to a perfect match between coders. We also computed P values for the null hypothesis H0: W = 0.

Intercoder agreement measures the degree to which different coders respond in the same way to a certain item.⁶⁰ In the case of two coders, interrater agreement pertains to the proportion of messages coded with the same response. Since we had ordinal items and three coders, we computed the average quadratically weighted Cohen's kappa⁶² and Krippendorff's alpha.⁶³ The values above 0.20 were understood as acceptable.⁶⁴ In Table 3, we report all intercoder reliabilities and agreements.

Table 3.

Interrater agreements and reliabilities.

Dimension	Item	Weighted Cohen's kappa	Krippendorff's alpha (ordinal)	Kendall's W	ICC (reliability)	Note
Readability	QISS1	.19	.15	0.44 (P < .001)	.43 [.32;.52]	Excluded
Readability	QISS2	.16	.19	0.47 (P < .001)	.37 [.26;.47]	Excluded
Readability	QISS3	.18	.16	0.44 (P < .001)	.39 [.27;.48]	Excluded
Readability	QISS4	.24	.23	0.50 (P < .001)	.49 [.40;.57]
Readability	QISS5	.35	.33	0.55 (P < .001)	.62 [.55;.68]
Readability	QISS6	.15	.16	0.45 (P < .001)	.35 [.23;.45]	Excluded
Readability	QISS7	.23	.18	0.47 (P < .001)	.50 [.41;.58]	Excluded
Readability	QISS8	.18	.15	0.44 (P < .001)	.41 [.30;.50]	Excluded
Accuracy	QISS9	.65	.56	0.71 (P < .001)	.86 [.83;.88]
Accuracy	QISS10	.70	.61	0.74 (P < .001)	.88 [.86;.90]
Accuracy	QISS11	.68	.55	0.72 (P < .001)	.87 [.85;.89]
Accuracy	QISS12	.64	.52	0.71 (P < .001)	.85 [.82;.88]
Completeness	QISS13	.43	.38	0.63 (P < .001)	.73 [.69;.78]
Completeness	QISS14	.16	.12	0.47 (P < .001)	.40 [.29;.50]	Excluded
Completeness	QISS15	.32	.24	0.59 (P < .001)	.67 [.61;.72]
Completeness	QISS16	.37	.36	0.59 (P < .001)	.66 [.60;.72]
Completeness	QISS17	.33	.30	0.58 (P < .001)	.64 [.57;.70]
Completeness	QISS18	.66	.66	0.79 (P < .001)	.86 [.84;.89]
Completeness	QISS19	.48	.45	0.67 (P < .001)	.75 [.71;.79]
Completeness	QISS20	.39	.17	0.47 (P < .001)	.66 [.59;.71]	Excluded
Completeness	QISS21	.33	.16	0.52 (P < .001)	.66 [.59;.71]	Excluded
Argumentation	QISS22	.47	.46	0.63 (P < .05)	.72 [.28;.91]
Argumentation	QISS23	.49	.49	0.60 (P < .05)	.61 [.00;.87]
Argumentation	QISS24	.62	.54	0.76 (P < .001)	.84 [.60;.95]
Argumentation	QISS25	.21	.19	0.44 (P = .21)	.48 [−.37;.84]	Excluded
Trustworthiness	QISS26	.70	.58	0.72 (P < .001)	.89 [.87;.90]
Trustworthiness	QISS27	.29	.29	0.53 (P < .001)	.57 [.49;.64]
Trustworthiness	QISS28	.39	.26	0.69 (P < .001)	.76 [.72;.80]
Trustworthiness	QISS29	.43	.32	0.66 (P < .001)	.75 [.70;.79]
Trustworthiness	QISS30	.67	.61	0.74 (P < .001)	.86 [.84;.88]
Trustworthiness	QISS31	.30	.29	0.54 (P < .001)	.52 [.43;.59]

We excluded some items from further analysis on the basis of the following criteria: if an item had a value of the ICC lower than 0.5 or if Kendall's W was not statistically significantly greater than zero at a 5% significance rate or if Cohen's kappa value or the Krippendorff alpha was lower than.2. Consequently, we excluded items QISS1, QISS2, QISS3, QISS6, QISS7, QISS8, QISS14, QISS20, QISS21, and QISS25 (see Table 3). We decided not to exclude item QISS4, where the value of ICC is just below the threshold (0.49), while the other three parameters are at least satisfactory. Unsurprisingly, many items regarding readability elicited low interrater reliability as this is a highly subjective dimension and is less dependent on the professional background of coders. The remaining items demonstrated at least satisfactory interrater agreement and interrater reliability.

Testing the multidimensional structure of the QISS measure. In the next step, we aimed to confirm the five-dimensional structure of the QISS in OHCs measurement instrument. From this test, we excluded the dimension of justification as the number of valid responses for this dimension was much smaller than for the other four dimensions (considering the fact that justification was only measured for messages where some sort of justification was present). Consequently, we tested the fit of the four-factor structure of the QISS measurement instrument. For this purpose, we used the CFA approach—package lavaan 0.7 in R⁶⁵ with a diagonally weighted least squares estimator, which is suggested in the case of ordinal scales. Since the four-dimensional model did not provide a good fit to the data, we ran an exploratory factor analysis (EFA) and discovered that the dimension of completeness was divided into two dimensions. One dimension pertained to all relevant aspects of the topic under discussion (we named this relevance), while the other dimension pertained to the extent to which the message comprehensively covered the facts regarding the topic under discussion and introduced new information (we named this breadth).

We reran a CFA with the five-dimensional structure, and in this case, the model fit was satisfactory (CFI = .99, TLI = .99, RMSEA = .04, SRMR = .10). Since the factor weights of two items (QISS26 and QISS27) were lower than 0.5, we excluded them from the factor model and reran the CFA.

The modified model demonstrated a good fit to the data (CFI = 1, TLI = 1, RMSEA = .01, SRMR = .05) (see Table 4). The measurement quality was also supported by Cronbach's alpha, which showed good to excellent internal consistency for all variables: readability (alpha = .84), accuracy (.98), relevance (.96), breadth (.84), and trustworthiness (.89). Conducting a CFA on the single dimension of justification was meaningless, but the EFA showed high factor weights (close to 1) supported by a high internal consistency of the scale (alpha = .95).

Table 4.

Confirmatory factor analysis (CFA) of the QISS scale.

Scale items	Accuracy	Readability	Relevance	Breadth	Trustworthiness	Justification
QISS4: Information is presented in layman's terms		.89
QISS5: Message contains too much medical jargon (R)		.82
QISS9: Information is congruent with valid medical guidelines	.94
QISS10: Information or advice is scientifically accurate	.94
QISS11: Information is made up (R)	.96
QISS12: Information doesn't have any scientific backing (R)	.96
QISS13: All relevant aspects of the health issue are presented			.99
QISS15: Relevant positive and negative aspects of the topic are presented			.98
QISS16: Some important information regarding the topic is missing (R)			.87
QISS17: New information regarding the discussed topic has been published				.76
QISS18: The message contains a wide spectrum of facts				.77
QISS19: The message only presents one aspect of the topic (R)				.89
QISS28: It seems that the message is biased by the author's commercial interests (R)					.90
QISS29: Information is presented in an unbiased way					.92
QISS30: Information is presented in a sensationalist way (R)					.85
QISS31: It seems that the author is misleading the audience (R)					.59
QISS22: Arguments are convincing						.91
QISS23: Arguments are supported with scientific evidence about cancer						.91
QISS24: Arguments are in line with the majority medical opinion on this issue						.98
Cronbach's alpha	.98	.84	.96	.84	.89	.95

Note: Dimension of justification was tested separately with an EFA.

The correlation matrix between the QISS dimensions (see Table 5) shows that the dimensions of accuracy, relevance, trustworthiness, and justification correlated strongly with each other. However, readability showed weak negative correlations with other dimensions. This suggests that more accurate, trustworthy, justified, and wide messages tended to be less readable. The two dimensions of comprehensiveness were moderately correlated with each other, while relevance seemed to be more strongly correlated with other QISS dimensions than breadth.

Table 5.

Correlation matrix between all QISS dimensions.

	Accuracy	Readability	Relevance	Breadth	Trustworthiness	Justification
Accuracy	1
Readability	−.14*	1
Relevance	.76*	−.09	1
Breadth	.20*	−.17*	.34*	1
Trustworthiness	.78*	−.13*	.70*	.18*	1
Justification	.91*	−.22	.82*	.24	.74*	1

Note: * P < .001.

Methods of analysis. To analyze the main research question, we used a one-way analysis of variance and the post-hoc Tukey HSD test for multiple paired comparisons. The F-test was used to investigate the statistical significance of the differences among the three types of forums, except for the dimensions of breadth and justification, where Welch's test was used due to the heterogeneity of variance.

Results

In this section, we present the results regarding the QISS in the OHC under study (i.e. MON) and the comparisons across the different types of discussion forums. For interpretation purposes, we also provide an additional table (Table 7), where indexes on a 1–5 scale are recoded in three values (e.g. accurate, indefinite, and inaccurate for the Accuracy dimension), similar to work done by researchers using the DISCERN tool (e.g. Haragan et al.⁶⁶).

Table 7.

Differences among types of forums in the percentages of messages with different QISS levels.

	Values	Total	Socializing forums	Online support group forums	Online consultation forums
Readability	Unreadable	1.2%	0%	0%	1.6%
	Indefinite	4.0%	5.0%	1.5%	4.4%
	Readable	94.8%	95.0%	98.5%	94.0%
Accuracy	Inaccurate	8.7%	23.5%	22.4%	4.7%
	Indefinite	10.0%	11.8%	11.9%	9.5%
	Accurate	81.3%	64.7%	65.7%	85.6%
Trustworthiness	Not trustworthy	4.5%	20.0%	10.3%	2.2%
	Indefinite	10.2%	35.0%	22.1%	6.0%
	Trustworthy	85.4%	45.0%	67.6%	91.7%
Relevance	Irrelevant	25.0%	40.0%	44.2%	18.7%
	Indefinite	19.9%	20.0%	17.6%	20.3%
	Relevant	55.1%	20.0%	38.2%	61.0%
Breadth	Narrow	35.2%	50.0%	38.2%	33.7%
	Indefinite	25.1%	35.0%	23.5%	24.8%
	Wide	39.7%	15.0%	38.2%	41.6%
Justification	Unjustified	25.0%	100.0%	40.0%	16.0%
	Indefinite	18.8%	0.0%	20.0%	20.0%
	Justified	56.3%	0.0%	40.0%	64.0%

The analysis of the MON OHC (See Table 6) shows that cancer-related messages providing informational social support had the highest level of readability (M = 4.67), followed by trustworthiness (M = 4.48) and accuracy (M = 4.30). Other dimensions were less developed, with the breadth of information (M = 3.13) and level of justification (M = 3.54) seeming most problematic. The results from Table 7 are even more telling: 94.8% of the messages were readable, 81.3% were accurate in terms of professional standards, 85.4% were trustworthy, 55.1% were relevant, 39.7% had enough breadth, and 56.3% provided quality arguments when there was some sort of argumentation present in the message. Noteworthy, only 8.9% of the messages contained some sort of justification for the statement provided.

Table 6.

Differences in the averages of individual QISS dimensions in cancer-related messages (n = 403) across forum type.

	Total		Socializing forums		Online support group forums		Online consultation forums		P value
	Mean	SD	Mean	SD	Mean	SD	Mean	SD
Readability	4.67	0.50	4.68	0.51	4.83	0.32	4.64	0.52	.001
Accuracy	4.30	1.07	3.62	1.19	3.69	1.43	4.48	0.89	<.001
Trust-worthiness	4.48	0.79	3.53	1.01	3.98	0.96	4.65	0.64	<.001
Relevance	3.41	1.16	2.45	1.04	2.85	1.26	3.59	1.08	<.001
Breadth	3.13	1.11	2.74	0.95	3.08	1.11	3.17	1.12	.226
Justification	3.54	1.36	1.67	0.94	2.93	1.34	3.81	1.26	.05

There were some notable differences in the assessed QISS and its dimensions across the types of forums in the OHC. The average readability, which was generally very high, was statistically significantly higher (P < .001) for messages in the online support groups than in the online consultations. Regarding accuracy, we observed high discrepancies across the types of forums as the average accuracy of the messages in the online consultations was significantly higher than in social support groups (P < .001) or the socializing forums (P < .001). Similar results can be seen in Table 7. In the online consultations, 85.6% of the messages were accurate in comparison with the socializing forums and support groups, where 64.7% and 65.7% of the messages were accurate, respectively. In terms of trustworthiness, the differences were also significant and large among the types of forums. The messages in the online consultations were most trustworthy (M = 4.65), significantly higher than in the online support groups (M = 3.98), which had significantly higher trustworthiness than in the socializing forums (M = 3.52). In the online consultations, 91.7% of the messages were identified as trustworthy, while only 45% of the messages in the socializing forums were identified as trustworthy. Similar differences can be observed regarding the dimension of relevance. The socializing forums had the least relevant messages (M = 2.45), while the messages in the online consultations were the most relevant (M = 3.59). Regarding the breadth of the information, there were differences within the sample of messages, but they were not statistically significant. In general, only 39.7% of the messages in all forums were identified as having sufficient breadth, with the socializing forums having only 15% of messages in this domain. Although the quality of justification was evaluated for only a small number of messages (n = 36), we were still able to note significant differences among the types of forums, with the online consultations having a much higher quality of justification (M = 3.81) than the online support groups (M = 2.93) and the socializing forums, which had a very low level of justification (M = 1.67).

Discussion

Principal findings

The main goal of this paper was to introduce the QISS concept and develop and validate its measurement scale in the context of MON, the largest OHC in Slovenia. The goal was also to investigate the level of QISS in MON in regard to cancer-related messages and the QISS differences among various types of discussion forums: online consultations, online support groups, and socializing forums.

From a methodological standpoint, the results demonstrate that the six-dimensional QISS scale comprising 20 items can be used as a reliable measure of QISS in cancer-related discussions in OHCs. Moreover, all the QISS items demonstrated at least a good level of interrater reliability and interrater agreement, proving the scale's usefulness for the content analysis of informational social support in OHC discussions.

Furthermore, the results show that it was meaningful to measure individual QISS dimensions. The informational social support provided in the MON OHC was not of equal quality in terms of its dimensions. More specifically, we found that readability was the most developed QISS dimension, followed by trustworthiness and accuracy, while the dimensions of relevance, breadth, and justification were less developed. In other words, in the studied OHC, MON, the information in cancer-related informational social support is provided in a way that is easily understood, it is provided by relatively trustworthy authors, moderately congruent with the standards and findings of medical professionals, and somewhat limited in completeness and the provision of evidence when argumentation is present in messages. These results suggest that cancer-related discussions in OHCs are not immune to the phenomenon of the problematic quality of information identified in other less interactive online spaces.^26,29

One of the main findings of this study points to the high importance of online consultations with health professionals in the context of OHCs. Only for this type of discussion space, it can be concluded that the quality of cancer-related informational social support is sufficiently high—a large majority of the messages were accurate, trustworthy, and relevant. Conversely, in the socializing forums and support groups, the accuracy and trustworthiness of the information in the messages were much lower. The significantly lower QISS score for the online support groups is very worrying, as studies suggest⁶⁷ that users expect credible information in such spaces, where cancer patients, cancer survivors, and members of cancer patient associations exchange information. Similarly, in the socializing forums, more than one-fifth of all messages in the online support groups were identified as inaccurate. This is a very problematic finding, especially considering the context of cancer, where every inaccurate message might lead to harmful consequences for the reader, who may be a patient or caregiver.^34,35 Although online support groups and socializing forums are important for users to receive emotional social support and find meaning and self-expression, they are also places where users receive informational social support.^68,69 The informational social support received in online support groups and socializing forums might be problematic in objective terms, but further investigation is needed to assess how different QISS levels affect how patients address their health-related needs, decision-making, disease management, interaction and communication with physicians, and even their health outcomes and well-being. Further research is also needed to evaluate how this impacts the future prospects of OHCs, as socializing forums are crucial for the sustainability of the online community³⁷ and the recruitment of new users. The question arises as to what happens when this “socializing base” of the forum is also a breeding ground for the exchange of informational social support of dubious quality. This is not to suggest that users automatically believe what is shared in such forums, as recent research suggests that OHC users utilize similar criteria as those constituting the QISS dimensions in judging the quality of information posted in discussion forums.³¹ However, since most of the active and vocal users who participate in socializing forums have the highest sense of belonging to the online community,⁷⁰ there is a risk of a strong echo chamber effect, where the same (low-quality) information is reproduced within a group of like-minded people.⁷¹ Thus, there is a paradoxical situation in OHCs where social support serves as a driver of belongingness and information sharing,^7,24 yet socializing forums, where users very often exchange social support, are also places where problematic QISS is exchanged.

Another rather problematic finding pertains to the fact that only 8.9% of all cancer-related messages contained some sort of justification for the posted statements, whether this was providing a link to some scientific publication or official source or providing rational arguments for the advice, referral, or information. This finding is not specific to the studied OHC, since studies^36,72 have recently discovered a complete absence of references to information sources. Although a low level of justification is expected in socializing forums and online support groups, as everyday chitchat rarely exhibits properties of deliberative communication,⁴² we would expect the presence of justification to be higher in online consultation forums as “expert patients” are often not satisfied with guidelines and tend to demand an additional explanation from health professionals. Furthermore, justification is one of the crucial components of rational communication in online spaces and needs to be cherished in order to avoid uncivil behavior,⁴² which has increasingly characterized content produced via user communication.

The cancer discussions in studied OHC lack completeness of information as the majority of the messages did not have enough breadth and/or lacked relevance to the topic under discussion, which is in line with findings that different aspects of information (such as risks) are infrequently reported in OHCs.²⁰ The lack of completeness of informational social support might be linked to the dominant format of communication in online communication, namely, that communication is highly dynamic and based mostly on short messages that do not allow one to present comprehensive aspects of a certain issue. To some extent, this finding is also connected to the methodological design as we focused on individual messages and not on whole discussion threads. However, we consciously made this decision as users could be exposed to messages without being aware of the context, as they arrive at the message via a search engine and not from reading the whole discussion thread.²³ It is difficult to expect that a single message will cover all perspectives of a certain question, and for this reason, it is important that discussion threads are designed in a way that users can easily access all relevant messages.

Surprisingly, the results of this study show that the readability of messages providing informational support was not at all problematic. In contrast, a recent study on information regarding complementary and alternative medicine demonstrated that a great deal of information exchanges are characterized by poor readability.³⁶ The cancer-related messages in the studied OHC seemed understandable to lay people. Some studies suggest that health information provided by health professionals might be full of medical terminology,⁷³ but this did not appear to be a problem with the studied OHC. As previous research on the studied OHC showed,¹² this might be due to the efforts that health professional moderators invest in adopting medical terminology when explaining clinical terms, diagnoses, or specific indications of health conditions to OHC users. This readability of informational support is very important for patients’ understanding of information and their assessment of its credibility.⁷⁴

Research and practical implications

We believe that the introduction of the concept and measure of the QISS has important implications for research. Decades of research on informational social support in online spaces have been unable to control the quality of information exchanged in supportive relationships. However, researchers can now apply or adopt our coding scheme directly to different OHCs (for the coding scheme, see Appendix 1), whether they are composed of single-type discussion spaces, such as online support groups, or a conglomerate of different types of discussion forums. The coding scheme is applicable to all types of messages in online discussion spaces that arise in one-to-one, one-to-many, or many-to-many communication situations. In addition, the introduction of the QISS measure now allows researchers to empirically investigate how low-quality information sharing in online supportive communication affects individuals, the patient–physician relationship, and the health care (system) in general.

Moreover, we believe that the sampling methodology—comprised of the initial identification of all cancer-related messages with a machine-learning procedure for the sampling of the messages for coding—is also something that can be replicated and upgraded in further studies, not only of cancer-related discussions in OHCs but also other health issues. This way, it can become a standard procedure for identifying health-related messages in OHCs that provide informational social support and can be used to conduct manual content analysis for QISS measurements.

The coding scheme developed to measure the QISS in this study is primarily a research instrument and not a tool for consumers to evaluate the quality of information exchanged in OHCs. However, with some modification, our measurement instrument could also be used by consumers of information in OHCs but, more importantly, by OHC managers, who might be able to use it to detect QISS levels across discussion forums and topics, as this tool is not limited to cancer-related messages but can be adapted to any kind of health-related issue. This, of course, points to the need for expert coders with domain expertise, which might be overly costly for OHC managers. However, OHCs can attract other experts who are not medical professionals, in less costly ways. Namely, OHCs nowadays often collaborate with patient associations, patient advocacy groups, and experienced patients (e.g. Guillamon et al.⁷⁵), who have extensive knowledge about specific (chronic) health conditions and are an important source of credible health information and informational social support. Often, such actors in OHCs also take on the role of (discussion) moderators and could be placed in the role of the experts assessing and measuring the QISS in OHCs. Such participation is likely to be in the interest of many stakeholders as it would help them achieve the goals of increasing health literacy and promoting specific health information and approaches. Furthermore, a recent study demonstrated that the detection of some QISS aspects could be automated,¹⁸ suggesting that, in the future, OHCs will be able to implement algorithms to help attenuate the spread of low-quality information. A fruitful step in this direction would be the development of a machine learning algorithm trained to answer the questions in the coding scheme instead of experts. At the moment, we can only speculate about the efficiency of such an algorithm, but with good learning sets from various studies and with rapid advances in language processing models, we may soon be able to use artificial intelligence to replace expert coding.

Our study confirms that OHCs are faced with two main challenges regarding the QISS exchanged in forums. Although previous studies have suggested that (some) OHC users might lack adequate skills to assess, understand, and judge the information they receive,^74,76 this study amplifies the problem by showing that the objective QISS exchanged in forums is indeed questionable and that mechanisms should be put in place on both sides to increase the level of e-health literacy of users and attenuate the presence of low QISS in OHCs. OHC users should not only have the opportunity to participate in training to raise e-health literacy but also understand that the informational support received via OHCs should be used as complementary and in relation to the communication and information received from health professionals.⁶⁷ To some extent, this kind of complementarity can be enabled within the OHCs themselves as we discovered “safe spaces” with high QISS levels. It seems crucial that OHC users who receive informational support in online support groups or socializing forums would have the possibility to cross-validate this information in online consultation forums moderated by health professionals. Moreover, systems could be put in place where health professional moderators would be notified when informational social support related to their medical expertise is exchanged in socializing forums or online support groups and asked to verify or complement it. As mentioned earlier, there are machine learning approaches that quite efficiently identify different types of misinformation,¹⁹ and OHCs should try to implement such algorithms. They could, for example, automatically notify users about potential misinformation and/or call for the intervention of health professional moderators to validate or correct information. Ideally, health professionals would evaluate and approve a priori any kind of health-related informational support,⁶⁸ but this is not sustainable by current OHC business models. In any case, providing high-quality information in consultations with health professionals is also important for the continued use and sustainability of OHCs.⁷⁷

Cancer patients’ use of OHCs to fight their disease is unavoidable, and it is the responsibility of OHC managers to ensure that the informational support that they receive is based on high-quality information. A high presence of moderation of health professional forums and authors with expertise is a guarantee of high QISS. This also suggests that other parts of the OHC, which are primarily intended for chitchat but from time to time also touch on cancer-related topics or health issues in general, need stronger levels of moderation, at least for messages providing informational social support. For a start, such moderation could be assisted with the integration of machine learning algorithms such as those used in our study, which detected cancer-related discussions with a relatively high level of accuracy. Moreover, since cancer-related topics appear in very different forum threads that were not originally intended for cancer-related discussions, OHC managers should—with the help of health professionals—provide forums that explicitly address various topics related to cancer. We believe that the development of such mechanisms is a matter of urgency as the exchange of informational support goes far beyond the intended audience of OHCs.⁷⁸ Informational social support can also be important for lurkers and other Internet users. The content of OHCs is usually public and indexed in web search engines, thus making it accessible to anyone on the Internet. As OHCs present a public online source of health-related information, the QISS measurement is also a relevant tool for informing health promotion and education strategies and health policymakers. The QISS scale can be used to measure the QISS exchanged in various types of OHCs or other health-related apps and networks and can help experts identify specific information and topics that are misunderstood or deprived of explanation in the wider public or among specific (chronic condition) patients. This could be one of the important strategies to improve awareness, health literacy, and health-related knowledge among specific population segments about misconceived health topics and health-related behavior.

Limitations

This study introduced a novel concept, measurement instrument, and sampling methodology. However, it is also faced with certain limitations that arguably warrant further research. First, it is limited to a single OHC, thus presenting shortcomings not only in terms of generalizability but also in relation to the elements of OHCs, such as the type of moderation, the structure of forums, layout, and interactive features that might vary across OHCs, platforms, and national contexts. Consequently, the methodology used in this paper might not be directly applicable to other platforms and contexts that host OHCs. Nevertheless, the studied OHC is comparable to other known OHCs, such as PatientsLikeMe and MedHelp, and according to many Eurostat information society indicators, Slovenia is close to the median position among EU countries.⁷⁹ Consequently, the findings should inform beyond the OHC under study. However, the validation of the QISS scale should include further studies covering different samples, cultures, and OHC settings.

Second, the study was mostly descriptive and exploratory, which raises a number of questions that could not be examined within the limits of this paper. It would be necessary to investigate the question of the dominant factors influencing the QISS at both the individual and structural levels as online communities are complex sociotechnical systems where social actions result from both the sociotechnical structure and individual dispositions and practices.³⁷ Even more relevant are questions pertaining to the consequences of the QISS for the audience, patients, doctor–patient relationship, and health care in general. Most notably, a number of studies have identified informational social support as a driver of patient empowerment,^2,69 but can we speak of empowerment if the support is based on low QISS levels?

Third, the confirmation of the QISS scale is somewhat limited in terms of the justification dimension. As mentioned earlier, this dimension was tested separately on a small sample of messages, resulting in very strong correlations with the dimensions of accuracy and relevance. Further studies should look for ways to measure justification as a property of all messages, similar to other QISS dimensions.

Fourth, as already mentioned, the scale is primarily intended for use in research rather than in practice. However, we believe that OHCs could use the scale by engaging expert coders from partner organizations and/or develop the scale as a basis for developing artificial intelligence systems to assess the quality of information exchanged.

Conclusion

The problem of misinformation and other problematic aspects of information shared online does affect OHCs, at least to the same extent as in other online venues. Although fake news has an impact on global political systems,⁸⁰ it is not unlikely that low-quality information has an impact on an individual's health and on health systems in general. Especially in the context of COVID-19, we have witnessed how problematic social media, in general, can be in promoting low-quality information.⁸¹ However, instead of condemning online discussion platforms, we need to accept the fact that patients will always turn to online information sources. In this context, it is important to recognize that OHCs can be safeguards of high-quality information, but new models will need to be developed that will allow a higher level of integration of health professional moderators and the implementation of mechanisms to attenuate the sharing of low-quality information among users.