Abstract
Introduction
Peer online mental health forums are commonly used and offer accessible support. Positive and negative impacts have been reported by forum members and moderators, but it is unclear why these impacts occur, for whom and in which forums. This multiple method realist study explores underlying mechanisms to understand how forums work for different people. The findings will inform codesign of best practice guidance and policy tools to enhance the uptake and effectiveness of peer online mental health forums.
Methods and analysis
In workstream 1, we will conduct a realist synthesis, based on existing literature and interviews with approximately 20 stakeholders, to generate initial programme theories about the impacts of forums on members and moderators and mechanisms driving these. Initial theories that are relevant for forum design and implementation will be prioritised for testing in workstream 2.
Workstream 2 is a multiple case study design with mixed methods with several online mental health forums differing in contextual features. Quantitative surveys of forum members, qualitative interviews and Corpus-based Discourse Analysis and Natural Language Processing of forum posts will be used to test and refine programme theories. Final programme theories will be developed through novel triangulation of the data.
Workstream 3 will run alongside workstreams 1 and 2. Key stakeholders from participating forums, including members and moderators, will be recruited to a Codesign group. They will inform the study design and materials, refine and prioritise theories, and codesign best policy and practice guidance.
Ethics and dissemination
Ethical approval was granted by Solihull Research Ethics Committee (IRAS 314029). Findings will be reported in accordance with RAMESES (Realist And MEta-narrative Evidence Syntheses: Evolving Standards) guidelines, published as open access and shared widely, along with codesigned tools.
Trial registration number
ISRCTN 62469166; the protocol for the realist synthesis in workstream one is prospectively registered at PROSPERO CRD42022352528.
Keywords: MENTAL HEALTH, Patient Participation, QUALITATIVE RESEARCH, HEALTH ECONOMICS
Strengths and limitations of this study.
A detailed methodology is presented to develop and test a programme theory about how and why online forums can support people with mental health challenges and their carers. This methodology may inform the design of future studies to evaluate complex online interventions which are already in widespread use.
A novel triangulation of survey, interview and corpus linguistic methods is described, that integrates the analysis of primary and secondary data sources, requiring collaborative working between interdisciplinary researchers with varying levels of experience in realist approaches.
The process by which realist programme theory can be used to inform the codesign of best policy and practice guidance with people with lived expertise is outlined. This demonstrates the role of patient and public involvement in bridging the gap between theory development and intervention design.
Ethical issues of using online forum posts in research are identified, and a framework for addressing these will be produced.
Introduction
Mental health problems affect approximately 1 in six people in the UK at any one time (>11 million people).1 People seek support in a variety of ways, increasingly through peer support and by going online.2 3 They search for information and join online mental health forums: dedicated support platforms aimed at helping individuals to share, discuss and solicit information and support.4 Many are moderated, and some have access to additional online tools and resources. Although widely used, the impacts of using online forums, and mechanisms by which these occur, are poorly understood.
Online forums can provide round-the-clock opportunities for members to feel understood, access information, make friends and use their experiences to help others5 while being able to manage how they present themselves online.6–8 These forums can function as an important gateway to other offline support as people test out sharing their problems and are encouraged to access help.9–12 Online forums are particularly valued by people with rare or highly stigmatised difficulties, and those living in remote areas or close knit communities wanting anonymity.13 Online forums offer robust support during pandemics when healthcare services are restricted, and for people in low-income and middle-income countries where access to mobile technology is common but healthcare is not. Studies have shown positive outcomes associated with using online forums, including reduction in depression,14 suicidal thoughts,15 and social isolation.13
However, negative impacts of using online forums have also been reported including increases in suicidal ideation,16 more negative mood17 and worsening of body image and disordered eating patterns.18 Forums carry inherent risks of disinformation, dependency and sharing of unhelpful strategies. Information shared can be misleading, inaccurate, harmful or triggering.5 19 Hearing about the experiences of others can exacerbate low mood and risk of harms20 and generate unrealistic expectations and greater anxiety or confusion about one’s own condition.21 Normative ideas about health may leave some users feeling misunderstood or even bullied, and people in mental distress may be particularly vulnerable to the negative impacts of this kind of behaviour.22 Healthcare staff may be reluctant to moderate online forums because of concerns about possible harms and lack of protocols for online working,23 and may be reluctant to recommend specific forums without clearer guidance on criteria by which to make this decision.24
Current research suggests a wide range of impacts from using online mental health forums, but with little understanding of why these vary across individuals.13–19 Particular features of forums such as size and heterogeneity, target population, ease of access and level of friction, how individual identities are managed, forum rules, and incentives for use and for prosocial behaviour have all been hypothesised as features that might account for the variation in impacts.25–28 Style of moderation is also likely to be important. The moderator role is highly variable across forums, ranging from identifying and removing risk-related materials, to that of a highly skilled online counsellor, directing the focus and values of the online forum.29
Understanding how these contextual features influence the culture and function of forums is crucial. Hence, research needs to go beyond the study of behaviour in individual forums and investigate what aspects of design, moderation and culture differ between forums, which facilitate better outcomes and how.25 30 Research to date has drawn on quantitative surveys, qualitative interviews, computational discourse analysis and social network analysis, but all have limitations when used in isolation. Our research will move the field from single forum and single methods studies to the triangulation of findings from a range of methods and online forums, sampled for contextual diversity, to test comprehensive programme theories. The programme theories will then be used to inform codesign of best practice and policy guidance with the aim of maximising positive impacts of online forums, while minimising risks of any negative impacts on users and moderators.
Aims and objectives
Aims
To develop a programme theory to explain the underlying mechanisms by which online mental health forums impact on people’s mental health and well-being.
To use this programme theory to develop best practice tools to improve uptake, safety and usefulness of online forums.
Objectives
Objectives are aligned to specific research questions (RQ).
RQ1: What are the impacts of using online mental health forums for people experiencing mental health difficulties? How, why, in what contexts, and for whom are these impacts generated?
Objectives
1a. Develop theories of the underlying generative mechanisms by which, and contexts within which, online mental health forums impact on mental health and well-being outcomes for members.
1b. Test and refine the theories in case studies of online mental health forums.
RQ2. What are the roles of forum moderators and how do they impact online mental health forums?
Objectives
2a. Develop a theory to understand the roles of online mental health forum moderators, how they vary in different contexts and how they impact forums.
2b. Test and refine the theories in case studies of online mental health forums.
RQ3. How can evidence-based theories of online mental health forums be used to inform best practice guidance and support?
Objective
3a. Coproduce best practice tools, training and implementation plan to optimise the design and delivery of online mental health forums.
Methods and analysis
Study design
A realist informed mixed-methods evaluation, including multiple case studies31 to codesign theory informed best practice guidance tools for online mental health forums.
Date collection will start in March 2022 and finish in December 2024.
Figure 1 provides an overview of the study design.
Figure 1.
Study design of realist evaluation of online mental health communities.
Setting
Our research takes place online. Our cases are mental health forums, defined as:
Online spaces to facilitate peer-to-peer sharing, which are aimed at young people and adults in the UK, and set up by a UK-based healthcare provider, charity or commercial organisation with the explicit purpose of facilitating peer-to-peer support for people with mental health difficulties. Although people talk about mental health online in a variety of online spaces (including Facebook, Twitter, Instagram), we have focused on UK-based forums set up specifically to support mental health, because we hypothesise that the impacts of forums will be strongly influenced by their broader social context, including access to other health and social care services.
We aim to recruit 6–8 forums, purposively sampled for diversity across host, target population, design (including level and nature of moderation, and whether they require registered login) and size of population. Forums will be identified using snowballing techniques and online searching. We aim for maximum variation in our sample.
Patient and public involvement (PPI) statement
Prefunding, we hosted two online PPI events, each with 12 people with lived experience of moderating and/or regularly using online forums. Moderators reported significant challenges in their roles, impacting on their own mental health and were very keen to receive better training and support. Forum members identified the need for healthcare staff and commissioners to be more aware of the role online forums can play in mental health support, and the need to understand the role of ‘super-users’ who post a lot, and ‘observers’ who read but do not post.
Our project management team includes people with lived experience of using online resources to manage mental difficulties guiding the design and delivery of the project.
We will establish a PPI/Codesign group consisting of forum members and moderators from participating forums. We will recruit widely across age, gender, ethnicity, disability and digital engagement. The group will meet online monthly, facilitated by our PPI leads and will contribute to the design and materials for the study, prioritise and refine programme theories and codesign key outputs.
We also have two people with lived experience of using online mental health forums as independent members of our Study Steering Committee, meeting approximately every 6 months, to oversee the conduct of the study on behalf of the funder.
All PPI members will be invited to whole team training at the launch of the study to support a working understanding of all research methods drawn on in this study. Individual needs for training and support will be regularly reviewed and provided from across the team.
PPI will be reported in the final paper according to the GRIPP2 reporting checklist.32
Methods
The study is organised in three parallel workstreams.
Workstream 1 (1–18 months): theory building and hypothesis generation
Following established realist methodology,33–35 we will develop our initial candidate theories of how online mental health forums ‘work’ (RQ1a) by drawing on well-established theories36 reviewing relevant literature (including grey literature and intervention protocols), and conducting in-depth interviews with key stakeholders. The importance of forum moderation on outcomes is well established,5 13 29 37 but not well understood. Therefore, to answer RQ2a, we include a nested review question focussing specifically on understanding the role of the moderator and how this works in different contexts.
We will use the framing of If-then Context-Mechanism-Outcomes (CMO) configurations to guide all stages of the process, from theory construction to the development of data collection protocols and data analysis. Outcomes will include positive and negative impacts on forum member emotional well-being or functioning, such as anxiety, mood, personal recovery, quality of life and help seeking. Mechanisms will be defined as the responses of the forum members to the forum resources (including moderation) such as safety, toxicity, empathy, normalisation, bullying or reframing. Context will be explored at several levels and include any features of the forum, or backdrop against which it is being delivered, or the individual members, that may influence these mechanisms, for example, hosting, design, size, target population or anonymity.
We will first develop an initial framework for candidate CMOs based on existing formal theories. Selection will be guided by the extent to which each theory addresses our research aims, and their compatibility with the realist approach and underlying philosophy, that is, to what extent they help to identify generative causal explanations for key outcomes.
Next, with additional support from our information specialists at Lancaster University Library, we will develop a strategy to review existing literature in relevant electronic databases such as Medline, PsychInfo, EMBASE and additional grey literature (using Google, TRIP, overton.io, ProQuest, Allcatsrgrey, the International Clinical Trials Registry and the National Health Service (NHS) Knowledge and Library Hub) that report on the contexts in which online mental health forums are offered; the intended/proposed/measured outcomes and explicit/implicit theories about how they ‘work’. The search strategy will be systematic and clearly documented but will develop iteratively to include more diverse data sources that can shed light on generative causal pathways to outcomes. Data sources will include primary data, reviews, policy documents, training manuals, theoretical commentaries, author interpretations, websites and unpublished reports that give a rationale for offering online mental health forums, describe or evaluate online mental health forums or detail the lived experiences of members/moderators/hosts/commissioners. Screening (initially by title and abstract, with 10% checked by a second reviewer) will be based on functional criteria of relevance, rigour and richness, that is, whether the document tells us anything about contexts, mechanisms or outcomes of interest, rather than methodological quality.33 Data extraction will focus on extracting explanatory accounts and mapping where this data are coming from (the context, nature of the intervention, etc). In parallel to the literature review, we will interview up to 20 stakeholders including academics, clinicians, commissioners, policy makers and moderators with broad expertise in online mental health forums. Sampling will be done through snowballing techniques, drawing on the networks of our broad research team and key individuals identified in our literature searching. Data collection will be done flexibly (to maximise participation) using individual/group interviews, online/face to face/telephone and using topic guides designed to elicit CMO explanatory theories.
Analysis of literature and interview transcripts will be conducted using retroductive reasoning, that is, working back from the data to identify the context-dependent mechanisms underlying the impacts described in online forums. Particular attention will be paid to accounts of potential negative impacts (‘dark logic models’38) and those that are unexpected or challenge our initial programme theories.
We will articulate a theory of how peer online forums impact forum members and moderators, starting with an array of potential outcomes, multiple possible mechanisms and an initially broad context. We will work with our PPI/Codesign Group through prioritisation and consensus building to ensure that our final explanatory theory is comprehensible, testable and useful for the design of policy and practice tools in WS3.
Workstream 2 (months 6–30): theory testing and refinement
In workstream 2, hypotheses about (1) how online mental health forums work for whom, how and in what context (RQ1b) and (2) the role of forum moderators (RQ2b) will be tested in a longitudinal realist evaluation using novel triangulation of survey, interview and corpus linguistic methods.
We will first develop a detailed case report for each forum describing key contextual features (based on aggregate data provided by the host) identified in workstream 1 that influence user experiences, for example, design of forum (including access to additional online tools), ease of access and level of friction, size and heterogeneity of user population, how individual identities are managed (user identification vs anonymity), forum rules and incentives for use and for prosocial behaviour. We will describe moderation strategies in each case including how moderators are recruited, trained and supported, moderating rules and how they are developed, temporality, transparency and opportunities for redress to moderation decisions. We will describe the linguistic properties of each forum in these case descriptions, focussing on the differences between the forums in terms of theme, topics of discussion, semantic categories, sentiment and emotion categories and keywords.
Consistent with optimal case study design and realist approaches, we will take a mixed-method approach to explore relationships between quantitative measures of outcomes, contextual factors and some mediators, combined with an in-depth qualitative exploration of processes underpinning the deeper causal relationships between these variables. This will ensure that demiregularities between variables across the sample are not in themselves assumed to be causal.
Surveys assessing impacts within and across cases
Surveys of forum users will quantitatively assess impacts of forum use, and evaluate potential mediators and moderators, at baseline, 6 and 12 weeks. The choice of these intervals is based on times to clinically important change observed in previous online peer-support studies.39 40 Up to two iterations of the survey will allow for additional adaptation of measures should new hypotheses arise.
Participants will be recruited over a 9 month period across all forum cases, and will include members who passively view but rarely or never actively post to the forums (observers). Invitations and a link to the survey will be shared by forum hosts. On following that link, potential participants will see a participant information sheet (PIS) and consent form. After providing their consent, participants will complete the survey measures hosted on Research Electronic Data Capture (REDcap).41 To facilitate linkage between survey and forum use data, participants will be invited to provide their username if they wish, and to indicate if they are willing to be contacted for an in-depth interview. To ensure user anonymity, this linkage will only be for the period of the research and not available to moderators or other forum users.
We will adhere to British Psychological Society guidance42 for online informed consent including taking a record of valid consent; check boxes relating to specific consent statements; limiting the number of consent items and ensuring that participants were fully informed of study procedures, risks, confidentiality and right to withdraw.
The survey design will be guided by the programme theories. Where established measures of hypothesised impacts on forum member emotional well-being or functioning are available, these will be used, for example, GAD-7 for anxiety43 PHQ-8 for mood44 and where not, questions will be created.
Context will be measured at several levels and include any features of the forum, or backdrop against which it is being delivered, or the individual members, that may influence the extent to which hypothesised mechanisms are triggered. We will collect information on participant use of both NHS and privately funded healthcare services and medicines, in addition to community and third-sector services, related to their mental health, using an adapted online version of the Client Service Receipt Inventory23 and other validated resource use questionnaires.
All forum users over 16 years old will be eligible to take part in the survey, and we can compare demographic and usage data of the sample of those who do participate in the survey with the whole forum sample (where this data a available).
Analysis of survey data will include detailed descriptions of the forum samples and summary statistics on mental health outcomes. We will use the Mplus (V.8.6) software package45 46 to fit generalised mixed models and generalised structural equation models of the data, accounting for the hierarchical nature of the longitudinal data at the level of posts, users and forums. We used simulation results47 to estimate the sample size required to have statistical power of at least 0.8 to detect a small (a*b=0.02) longitudinal mediation effect across three waves of data. Assuming an Intra-Class Correlation of 0.6, a significance level of p=0.05 and 20% attrition per wave, we would need 602 participants, which is highly feasible across all forums.
Qualitative interviews
We will interview key stakeholders online including approximately 10–12 forum members, 4–5 moderators and 1–2 hosts or commissioners in each of our forum cases (total sample up to 114). This will enable us to test our hypotheses, while also being manageable within the resources of the project. Guided by realist interviewing methodology,48 49 theoretical sampling for forum members will be determined using an evolving sampling framework, drawing on our survey and analysis of forum posts to identify which participants are needed to test our programme theories. We will sample across observers, super-users and regular forum members, as well as people who have been invited to join an online forum but not done so.
Interview topic guides will include an opportunity for participants to share what they understand the role of online forums to be, their impacts and how they ‘work’. With our moderators, host and commissioners, we will also explore their understanding of the role of moderators, how they make decisions regarding moderating behaviours and how they are trained and supported in this role. Interview topic guides will evolve to specifically test our hypotheses. Interviews will be face to face or via secure video conferencing (Microsoft Teams), or telephone, recorded and transcribed.
Analysis will first be done within case, and then integrated to compare between cases. Data will be managed in NVIVO. Consistent with the realist approach, analysis will be retroductive, that is, seek to identify the hidden causal forces underlying people’s descriptions of their experiences in the forums. These will be initially coded into our hypothesised CMO configurations from workstream 1, but the analysis will be developed through regular discussion with the research team and PPI/Codesign group.
Analysis of forum posts
The collection of posts from our online forums will result in large linguistic data sets, potentially amounting to millions of words in some cases. To analyse these, we will use Corpus-based Discourse Analysis and Natural Language Processing.50 We will first describe the size, shape, content, emotional tone and linguistic features of each of our forum cases, also considering how each of these features change over time. In the second stage, we will adapt methods to test our specific CMO configurations. Specific techniques we can draw on to do this are summarised in table 1. Analyses will be carried out using the latest versions of cutting-edge corpus tools developed at Lancaster University (#LancsBox, the Lancaster Stats Tools Online, lexiDB and Wmatrix), and, where appropriate, the corpus analysis tools available in AntLab, and Sketch Engine and state-of-the-art NLP libraries and repositories such as Spacy and Hugging Face.
Table 1.
Overview of computational linguistic techniques to analyse forum posts
| Stage 1 technique | Function |
| Frequency and dispersion analyses | To obtain the frequencies and distribution of selected words, phrases or semantic domains in a corpus and its subsections. |
| Keyness analyses | To identify and study the words, phrases and concepts characteristic of a particular (sub-)corpus, for example, an online community vs a general corpus of English; moderated vs unmoderated communities; first posts vs last posts; patients’ posts vs moderators’ posts; etc. |
| Change over time analyses | To track changes in co-occurring words of critical interest over time to capture continuities and discontinuities in meanings and associations |
| Summarisation or topic modelling | To generate short summaries of forums or parts of forums and locate the most important posts. |
| Social network analysis | To extract the structure and shape of the network of people in the community, to see who is talking to whom, to classify user’s participation style (observers, users, super-users). |
| User profiling | To estimate the relative usage and representativeness of people in each community by age and gender, country location and other demographic variables |
| Stage 2 technique | |
| Concordance analyses | To obtain all occurrences of selected words, phrases or instances of different semantic domains, and to understand how and why they are used, for example, words such as ‘feel’, ‘daughter’ and ‘medication’ or semantic domains such as ‘Sad’, ‘Personal relationships’ and ‘Medicines and medical treatment’. |
| Collocation analyses | To study patterns of co-occurrence of words or phrases, which are well known to contribute to their meanings and associations, for example, what words tend to co-occur with ‘you’, ‘not’ or ‘worry’ in different (sub-)corpora. |
| Sentiment analysis | To study change in emotion over time and across forums or threads, showing positive and negative emotions and we will combine this with an analysis of risk indicator words and taboo words and terms of emotional support or bullying. |
Qualitative analyses will be employed to carry out in-depth studies of specific posts, interactions or threads selected to test our specific theories. This might include threads with particularly high or low frequencies of negatively balanced emotion words; posts or threads that were found to be prototypical of a whole forum in linguistic terms; posts that received large numbers of replies; posts by individuals representing different levels of engagement or roles with the online forum or showing evidence of significant change in emotional state. The qualitative analyses will investigate how users and moderators express and interactively manage identities, experiences, knowledge, emotions and relationships and with what local impacts in terms of patterns in responses and interactional behaviour. These analyses will be directed at testing our CMOs by exploring users’ understandings of mechanisms underlying the impact that participation in the online forum has on themselves and others.
Triangulation of data across methods and between cases
All methods have limitations, and in combining our survey, interviews and analysis of forum posts, we can ensure that we have a full range of perspectives (including those who choose not to take part in all aspects); we can purposively sample for interview on the basis of self-reported and real-time evidence of specific impacts, or membership to specific subgroups; we can move between broad contextual data, such as the size and structure of the forum, to detailed analysis of individual conversations: we can test specific programme theories in multiple ways to test reliability and validity of findings. Regular analysis meetings, including our PPI/Codesign group, will ensure that we make the most of the opportunity for iterative and integrated analysis across our cases and between our researchers using different methods.
Workstream 3 (months 2–34) impact
Our theories will be used to underpin the codesign of best-practice tools and support for people involved in commissioning, hosting, moderating or using online mental health forums (RQ3). All tools will be informed by the programme theory developed in workstreams 1 and 2. The exact nature of the tools will depend on the outcomes of the codesign process, but our preapplication work with our PPI, hosts and impact groups has suggested the candidate tools described in table 2.
Table 2.
Candidate tools and implementation
| Tool | Aim |
| For people with mental health difficulties, referrers and commissioners | |
| Knowledge exchange video/animation tools | Widening access and promoting greater uptake through better understanding of the role of online mental health forums and how they work. These tools will describe how online forums work, possible benefits, for who and how to identity safe and supportive forums. |
| For forum moderators | |
| Community of practice | Building capacity through facilitating mutual engagement, joint enterprise and shared practice. Such support is crucial to continuing professional development (CPD), and to prevent compassion fatigue and burnout. Openly available to all the moderators involved in our study and providing a safe online space for the forum to evolve from the outset of the study. |
| E-learning curricula | To train and support moderators in reflexive practice including understanding the moderator role; ethics of moderation; encouraging activity; understanding mental health; spotting moments of change (introductory linguistic analysis); managing challenging situations; widening access and welcoming diversity; identifying and managing risks; looking after yourself; role of supervision and peer support; signposting; CPD. The content will draw on challenges described by moderators during our interviews, and practical case examples of ways to manage these. |
| For forum providers/hosts, commissioners and policy makers | |
| Best practice design principles for online mental health forums | To widen access, maximise positive impacts and minimise harms. This guidance will provide the following: (1) High level design principles to guide forum design and moderation. (2) Practical guidance for implementing design features to improve usefulness and safety of online forums. (3) Examples of best practice and case studies on forum design and moderation practices. |
| Resource requirements for implementation of best practice design principles based on the disaggregation of the programme theories into resources and reasoning | This will be based on the disaggregation of the programme theories into resources and reasoning and will provide information on the resource configuration necessary to provide safe and effective online community. |
| Standardised forum characterisation and evaluation framework | To support standardised evaluation and comparison of forums during commissioning |
| Methodological guidance on how to evaluate online mental health forums | To provide guidance for evaluation of efficacy and effectiveness |
| For further theory and practice development | |
| Programme theory | To generate a theory that is generalisable to online mental health forums and with relevance to understanding online peer support interventions across healthcare. |
We will design all outputs and our dissemination strategy to be accessible across age, gender, ethnicity, disability and the digital divide. We will write plain English summaries, translated into other languages where possible, and disseminate via non-digital platforms (in person, radio and print media) as well as digitally (websites, social media, etc).
Knowledge mobilisation
As the host site for this study, Berkshire Healthcare NHS Foundation Trust will take a lead role in ensuring that the best practice tools outputs will be easily discovered by our relevant audiences, openly accessible, free to use and updated and maintained beyond the life of the project. Ongoing feedback and revisions will be built into the design to ensure that these tools evolve as needed. We will also work with the Health Education England learning hub (https://learninghub.nhs.uk), NHS England, ORCHA (https://orchahealth.com) and Oxford AHSN to explore how these tools can be best designed and promoted across moderators working across NHS, charity and commercial organisations.
Discussion
Online mental health forums have the potential to greatly increase access to peer support, but the impacts of using forums and the underlying mechanisms need to be better understood. This protocol outlines a methodologically novel triangulation of survey, interview and corpus linguistic methods, to generate and test realist programme theories about how online forums ‘work’ for forum users and moderators, and to use these theories to develop best policy and practice tools to improve peer online forums. This study will be an inherently iterative process, and therefore the aim of the protocol is not to prospectively define all our decisions at the outset, but rather to explore some of the key issues that have arisen and to share our learning from the challenges we have faced in designing the protocol.
The first challenge was to develop an ethical framework for the use of online forums posts as research data. We adopted a user-centred approach, considering a user’s likely expectations and concerns; consulted health forum hosts, moderators and users; hosted an online focus group (n=21) exploring the study’s benefits and risks and how best to mitigate these; examined key guidance on using online data including (a) the British Psychological Society42 and (b) Association of internet researchers;51 collaborated with legal experts at Berkshire Healthcare NHS Foundation Trust and the information officer at Lancaster University, and discussed our strategy widely to learn how other researchers in the field have managed these ethical issues. The outcome of this process is described in our data processing plan shown in table 3 and published on our study website (http://www.lancaster.ac.uk/ipof/ethics-framework). The process raised several interesting ethical issues which will be described in detail in a separate paper.
Table 3.
Data processing plan for iPOF
| Forum data | Survey data | Interview data | |
| Collection | Posts from publicly available forums with no expectation of privacy, where the forum host consents, will be accessed directly. Prior to downloading the data, the study will be described and users invited to withdraw their data if they wish. Posts from forums requiring a login, with an expectation of privacy, will only be used from individuals who have freely consented (ie, they had the option to not consent and still use the service) to their data being used for research. Prior to downloading the data, the study will be described and users invited to have their data removed if they wish. Posts from forums in which consent for data to be used for research is required before joining the forum will be used, but only where it is possible to ensure that all users have been made aware of the option to opt out. Posts from forums linked to health or social care records, and no consent has been given for research at sign up, will only be used with additional individual informed consent. Finally, posts from forums that are closed, publicly available and where there are no links between posts and any personal identifiable data will be used without additional consent. |
Using REDCap with individual informed consent. | Using Microsoft Teams and an encrypted audio recorder, with individual informed consent |
| Deidentification | When forums are anonymous and publicly available, we will replace all usernames with a personal identification number (PIN) and automatically remove any names of places or people that could be identifiable. For all other forums, posts will be deidentified by the host organisation before being shared with us. |
Identifiable data, such as on consent forms, will be stored separately to the survey results. | Identifying information will be removed from transcripts. Identifiable data, such as on consent forms, will be stored separately. |
| Transfer | Data transfer from online communities to Lancaster University will be secure and encrypted (eg, secure FTP and HTTPS). | Data will be collected via a link directly into Lancaster University system. | Audio files will be uploaded to Lancaster’s servers and deleted from the recorder. Transcription will be done by a University approved and contracted transcriber. |
| Storage | Using Lancaster University’s approved IT systems. | Data will be stored on Lancaster University’s secure servers. | Recordings and transcripts will be stored on Lancaster University’s secure servers. |
| Analysis | Analysis (Natural Language Processing) will be conducted by methodological expert members of the research team (see table 1). | Analyses will include detailed description of the sample and the use of generalised mixed models and structural equation models using Mplus (V.8.6). | Analysis will be retroductive and will contribute to hypothesised CMO configurations. Analysis will be managed in NVIVO. |
| Deletion | Participants will have the right to withdraw and request that their data be deleted, up to the analysis starting. | Participants have 1 week to request their data to be removed. | Participants will have the right to withdraw and request that their data be deleted, up to the analysis starting. |
| Publication | To preserve anonymity, paraphrased forum quotes will be published. | Minimum cell sizes will be adopted for published results. | If consent is given, direct quotes will be published. Any potentially identifying information will be removed. |
| Archiving and access | All papers will be published open access. Given the nature of the data we will not share the forum data sets openly. | Deidentified survey data will be openly shared on Pure. | Interview data will be restricted access and available by request to legitimate research parties if the purpose is consistent with the consent given for this research. |
CMO, Context-Mechanism-Outcomes; iPOF, Improving Peer Online Forums; REDCap, Research Electronic Data Capture.
The second challenge was to build a team of collaborators, each expert in their own discipline, but not all familiar with the realist approach. Statistical analysis of surveys, reflexive thematic analysis of interview transcripts, corpus linguistic analysis of forum posts and realist synthesis of published literature, all originate from different philosophical and theoretical positions, and vary in the extent to which they use different styles of reasoning (inductive, deductive, abductive and retroductive). We have divided the project into workstreams which focus initially on the set up and descriptive analysis of the survey, interview and linguistic data. However, we have also built a collaborative workstream throughout the project to focus on developing a shared understanding of the realist approach, to codevelop the programme theories and codesign novel integrated methods to test these.
The final challenge has been establishing our PPI/Codesign group. We wanted to ensure moderators and forum members for all our participating forums had a significant ongoing role throughout the project. They have the expertise to ensure that the study is designed and conducted in a way that is engaging and acceptable to participants (PPI work), and the outputs are designed to be useful and effective (codesign participation). We have planned for this group to meet monthly throughout the study. However, despite increasingly detailed guidance from the National Institute for Health and Care Research about payment for involvement,52 this has been a challenge. Due to the extent and longevity of involvement, PPI members are required to be employees of the University, to share evidence of their right to work in the UK and navigate a bureaucratic contract-based payment process. These payment methods were not designed for the inherently flexible and iterative nature of PPI codesign work described above. Fear of losing access to benefits due to being deemed able to work as a result of getting involved in this kind of project has been another barrier to engagement. This can exclude those from more marginalised groups, whose voices need to be heard in this kind of research.
Ethics and dissemination
Ethical approval was granted by Solihull Research Ethics Committee (IRAS 314029) on 20 June 2022. Table 3 describes each step in the process from data collection to dissemination for each of the three different types of data being collected.
We will define our range of audiences, identify what each needs to know and prepare targeted outputs appropriately created and written for this audience including Plain English summaries. Findings will be reported in accordance with RAMESES53 guidelines, published as open access and shared widely, along with codesigned tools. Study outputs will be promoted through a national launch of the codesigned tools, as part of a conference, webinar and social media strategy. Our protocols and academic findings will be published in open access peer-reviewed journals and presented at academic and peer forum conferences. All study participants and relevant health, social, educational and academic organisations will be kept updated via a study website (https://www.lancaster.ac.uk/ipof), social media (twitter) and blogs from across the wider team. All policy and practice guidance will be hosted, updated and made widely available via Berkshire Healthcare NHS Foundation Trust.
Supplementary Material
Footnotes
Twitter: @fionalobban, @kmachin, @RachelMeacock, @elenasemino
Contributors: All authors contributed to drafting the manuscript and approved the final version. The study was conceived by FL, developed for funding by FL, JR-M, ES, NS, PR, SHJ, CL, RM, SW, MR and further adapted in the setup phase by input from all authors. Theory development and testing is led by PM, HR and MC, with guidance in realist methodology from JR-M. Qualitative analysis of interviews and corpus linguistics is led by ES; computational analysis by PR and MC; survey design and analysis by NS; codesign by SHJ and KM and PPI by CL. The programme is managed by TR, with research administration support from RJ, ZG, KP and ED. Clinical input is provided by SHJ, FL, SW and JH.
Funding: This study is funded by the NIHR Health and Social Care Delivery Research (HS&DR) (NIHR134035). The views expressed are those of the author(s) and not necessarily those of the NIHR or the Department of Health and Social Care. The study is hosted by Berkshire Healthcare NHS Foundation Trust. The Sponsor is Lancaster University. Additional support comes from the NIHR Clinical Research Network, NIHR North-West Coast Applied Research Collaboration (ARC) and Oxford Allied Health Sciences Network (AHSN).
Competing interests: None declared.
Patient and public involvement: Patients and/or the public were involved in the design, or conduct, or reporting or dissemination plans of this research. Refer to the Methods and analysis section for further details.
Provenance and peer review: Not commissioned; peer reviewed for ethical and funding approval prior to submission.
Ethics statements
Patient consent for publication
Not required.
References
- 1.McManus S, Bebbington PE, Jenkins R, et al. Mental health and wellbeing in England: the adult psychiatric morbidity survey 2014. NHS digital, 2016. [Google Scholar]
- 2.NHS Confederation . Digital inclusion in mental health - a guide to help increase choice and improve access to digital mental health services; 2020.
- 3.Children and parents: media use and attitudes report 2019. London, UK: Ofcom; 2020. [Google Scholar]
- 4.Saha K, Sharma A, eds. Causal factors of effective psychosocial outcomes in online mental health communities. In: Proceedings of the international AAAI conference on web and social media. 2020. [Google Scholar]
- 5.Hanley T, Prescott J, Gomez KU. A systematic review exploring how young people use online forums for support around mental health issues. J Ment Health 2019;28:566–76. 10.1080/09638237.2019.1630725 [DOI] [PubMed] [Google Scholar]
- 6.Naslund JA, Aschbrenner KA, Marsch LA, et al. The future of mental health care: peer-to-peer support and social media. Epidemiol Psychiatr Sci 2016;25:113–22. 10.1017/S2045796015001067 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Coulson NS, Bullock E, Rodham K. Exploring the therapeutic affordances of self-harm online support communities: an online survey of members. JMIR Ment Health 2017;4:e44. 10.2196/mental.8084 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Brady E, Segar J, Sanders C. "I always vet things”: navigating privacy and the presentation of self on health discussion boards among individuals with long-term conditions. J Med Internet Res 2016;18:e274. 10.2196/jmir.6019 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Griffiths KM, Reynolds J, Vassallo S. An online, moderated peer-to-peer support bulletin board for depression: user-perceived advantages and disadvantages. JMIR Ment Health 2015;2:e14. 10.2196/mental.4266 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Powell J, McCarthy N, Eysenbach G. Cross-sectional survey of users of Internet depression communities. BMC Psychiatry 2003;3:19. 10.1186/1471-244X-3-19 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Prescott J, Rathbone AL, Hanley T. Online mental health communities, self-efficacy and transition to further support. Ment Health Rev J 2020;25:329–44. 10.1108/MHRJ-12-2019-0048 [DOI] [Google Scholar]
- 12.Bartlett YK, Coulson NS. An investigation into the empowerment effects of using online support groups and how this affects health professional/patient communication. Patient Educ Couns 2011;83:113–9. 10.1016/j.pec.2010.05.029 [DOI] [PubMed] [Google Scholar]
- 13.Smith-Merry J, Goggin G, Campbell A, et al. Social connection and online engagement: insights from interviews with users of a mental health online forum. JMIR Ment Health 2019;6:e11084. 10.2196/11084 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Griffiths KM, Mackinnon AJ, Crisp DA, et al. The effectiveness of an online support group for members of the community with depression: a randomised controlled trial. PLoS One 2012;7:e53244. 10.1371/journal.pone.0053244 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 15.Marchant A, Hawton K, Stewart A, et al. A systematic review of the relationship between internet use, self-harm and suicidal behaviour in young people: the good, the bad and the unknown. PLoS One 2017;12:e0181722. 10.1371/journal.pone.0181722 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Dunlop SM, More E, Romer D. Where do youth learn about suicides on the internet, and what influence does this have on suicidal Ideation J Child Psychol Psychiatry 2011;52:1073–80. 10.1111/j.1469-7610.2011.02416.x [DOI] [PubMed] [Google Scholar]
- 17.Moore D, Ayers S. Virtual voices: social support and stigma in postnatal mental illness internet forums. Psychol Health Med 2017;22:546–51. 10.1080/13548506.2016.1189580 [DOI] [PubMed] [Google Scholar]
- 18.Rodgers RF, Lowy AS, Halperin DM, et al. A meta‐analysis examining the influence of pro‐eating disorder websites on body image and eating pathology. Eur Eat Disord Rev 2016;24:3–8. 10.1002/erv.2390 [DOI] [PubMed] [Google Scholar]
- 19.Bailey E, Robinson J, Alvarez-Jimenez M, et al. Moderated online social therapy for young people with active suicidal ideation: qualitative study. J Med Internet Res 2021;23:e29645. 10.2196/29645 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Easton K, Diggle J, Ruethi-Davis M, et al. Qualitative exploration of the potential for adverse events when using an online peer support network for mental health: cross-sectional survey. JMIR Ment Health 2017;4:e49. 10.2196/mental.8168 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Ziebland S, Wyke S. Health and illness in a connected world: how might sharing experiences on the Internet affect people’s health Milbank Q 2012;90:219–49. 10.1111/j.1468-0009.2012.00662.x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Malik S, Coulson NS. They all supported me but I felt like I suddenly didn't belong anymore’: an exploration of perceived disadvantages to online support seeking. J Psychosom Obstet Gynaecol 2010;31:140–9. 10.3109/0167482X.2010.504870 [DOI] [PubMed] [Google Scholar]
- 23.Lobban F, Appelbe D, Appleton V, et al. Implementation of an online relatives' Toolkit for psychosis or bipolar (IMPART study): Iterative multiple case study to identify key factors Impacting on staff uptake and use. BMC Health Serv Res 2020;20:219. 10.1186/s12913-020-5002-4 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Lobban F, Appelbe D, Appleton V, et al. An online supported self-management toolkit for relatives of people with psychosis or bipolar experiences: the IMPART multiple case study. Health Serv Deliv Res 2020;8:1–248. 10.3310/hsdr08370 [DOI] [PubMed] [Google Scholar]
- 25.Krasodomski-Jones A. Everything in moderation: platforms, communities and users in a healthy online environment.
- 26.Sibilla F, Mancini T. I am (not) my avatar: a review of the user-avatar relationships in massively multiplayer online worlds. Cyberpsychology 2018;12. 10.5817/CP2018-3-4 [DOI] [Google Scholar]
- 27.Suler J. The online disinhibition effect. Cyberpsychol Behav 2004;7:321–6. 10.1089/1094931041291295 [DOI] [PubMed] [Google Scholar]
- 28.Smith J, Krasodomski-Jones A, Olanipekun M, et al. A picture of health March 2021: measuring the comparative health of online spaces; 2021.
- 29.Grimmelmann J. The virtues of moderation,17. Yale JL & Tech, 2015: 42. [Google Scholar]
- 30.Smith J, Jones E, Judson E. What’s in A name? A forward view of anonymity online.
- 31.Yin RK. Case study research and applications: design and methods. Sage publications, 2017. [Google Scholar]
- 32.Staniszewska S, Brett J, Simera I, et al. GRIPP2 reporting checklists: tools to improve reporting of patient and public involvement in research. BMJ 2017;358:j3453. 10.1136/bmj.j3453 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33.Pawson R, Greenhalgh T, Harvey G, et al. Realist synthesis: an introduction. Manchester: ESRC Research Methods Programme, University of Manchester, 2004. [Google Scholar]
- 34.Rycroft-Malone J, McCormack B, Hutchinson AM, et al. Realist synthesis: illustrating the method for implementation research. Implement Sci 2012;7:33. 10.1186/1748-5908-7-33 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35.Jagosh J. Realist synthesis for public health: building an ontologically deep understanding of how programs work, for whom, and in which contexts. Annu Rev Public Health 2019;40:361–72. 10.1146/annurev-publhealth-031816-044451 [DOI] [PubMed] [Google Scholar]
- 36.Hunter R, Gorely T, Beattie M, et al. Realist review. Int Rev Sport Exerc Psychol 2022;15:242–65. 10.1080/1750984X.2021.1969674 [DOI] [Google Scholar]
- 37.Wadden D, August T, Li Q, et al. The effect of moderation on online mental health conversations. arXiv 2020. 10.48550/arXiv.2005.09225 [DOI] [Google Scholar]
- 38.Bonell C, Jamal F, Melendez-Torres GJ, et al. Dark logic’: theorising the harmful consequences of public health interventions. J Epidemiol Community Health 2015;69:95–8. 10.1136/jech-2014-204671 [DOI] [PubMed] [Google Scholar]
- 39.Kaylor-Hughes CJ, Rawsthorne M, Coulson NS, et al. Direct to public peer support and E-therapy program versus information to aid self-management of depression and anxiety: protocol for a randomized controlled trial. JMIR Res Protoc 2017;6:e231. 10.2196/resprot.8061 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 40.Powell J, Hamborg T, Stallard N, et al. Effectiveness of a web-based cognitive-behavioral tool to improve mental well-being in the general population: randomized controlled trial. J Med Internet Res 2013;15:e2. 10.2196/jmir.2240 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 41.Patridge EF, Bardyn TP. Research electronic data captrue (REDCap). J Med Libr Assoc 2018;106:142–4. 10.5195/jmla.2018.319 [DOI] [Google Scholar]
- 42.Hewson C, Buchanan T, eds. Ethics guidelines for internet-mediated research. The British Psychological Society, 2013. [Google Scholar]
- 43.Spitzer RL, Kroenke K, Williams JBW, et al. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med 2006;166:1092–7. 10.1001/archinte.166.10.1092 [DOI] [PubMed] [Google Scholar]
- 44.Kroenke K, Spitzer RL, Williams JB. The PHQ‐9: validity of a brief depression severity measure. J Gen Intern Med 2001;16:606–13. 10.1046/j.1525-1497.2001.016009606.x [DOI] [PMC free article] [PubMed] [Google Scholar]
- 45.Maxwell SE, Cole DA. Bias in cross-sectional analyses of longitudinal mediation. Psychol Methods 2007;12:23–44. 10.1037/1082-989X.12.1.23 [DOI] [PubMed] [Google Scholar]
- 46.Rabe-Hesketh S, Skrondal A, Pickles A. Generalized multilevel structural equation modeling. Psychometrika 2004;69:167–90. 10.1007/BF02295939 [DOI] [Google Scholar]
- 47.Pan H, Liu S, Miao D, et al. Sample size determination for mediation analysis of longitudinal data. BMC Med Res Methodol 2018;18:32. 10.1186/s12874-018-0473-2 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 48.Pawson R. Theorizing the interview. Br J Sociol 1996;47:295. 10.2307/591728 [DOI] [Google Scholar]
- 49.Manzano A. The craft of interviewing in realist evaluation. Evaluation 2016;22:342–60. 10.1177/1356389016638615 [DOI] [Google Scholar]
- 50.Baker P, Gabrielatos C, KhosraviNik M, et al. A useful methodological synergy? Combining critical discourse analysis and corpus linguistics to examine discourses of refugees and asylum seekers in the UK press. Discourse Soc 2008;19:273–306. 10.1177/0957926508088962 [DOI] [Google Scholar]
- 51.Research AoI . Ethics: association of internet research. n.d. Available: https://aoir.org/ethics/
- 52.NIHR . Payment guidance for researchers and professionals; 2021.
- 53.The RAMESES projects. 2022. Available: https://www.ramesesproject.org/
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.