Open Science Practices in Psychiatric Genetics: A Primer

Abstract

Open science ensures that research is transparently reported and freely accessible for all to assess and collaboratively build on. Psychiatric genetics has led among the health sciences in implementing some open science practices in common study designs, such as replication as part of genome-wide association studies. However, thorough open science implementation guidelines are limited and largely not specific to data, privacy, and research conduct challenges in psychiatric genetics. Here, we present a primer of open science practices, including selection of a research topic with patients/nonacademic collaborators, equitable authorship and citation practices, design of replicable, reproducible studies, preregistrations, open data, and privacy issues. We provide tips for informative figures and inclusive, precise reporting. We discuss considerations in working with nonacademic collaborators and distributing research through preprints, blogs, social media, and accessible lecture materials. Finally, we provide extra resources to support every step of the research process.

Open science is an approach where research and resulting publications are transparent and freely accessible. Consequently, findings may be replicated, and errors prevented or corrected (1). The core of open science is a commitment to sharing research with everyone (2). Thus, several definitions of open science also address collaboration, inclusion, and diversity to improve participation in science (3,4). Consistent with this multifaceted view, we discuss open science practices related to transparency, integrity, diversity, inclusivity, equity, reproducibility, rigor, and accessibility (see Box S1 for definitions).

Psychiatric genetics has been praised as an early adopter of open science, championing genome analyses with embedded replication, initially through genome-wide association studies (GWASs) (5,6). Multiple consortia now exist to improve data pooling and provide free bioinformatic methods, e.g., the Psychiatric Genomics Consortium (PGC) (6), the Genotype-Tissue Expression Project (7), and the Global Biobank Meta-analysis Initiative (8). Geneticists frequently use preprint servers (where manuscripts get freely deposited before peer review), e.g., bioRxiv (9). All these efforts are aimed at remedying years of unreplicable research (10,11) due to small samples that were underpowered to detect associations between psychiatric disorders (12), singular gene variants, and environmental factors (13); questionable statistical practices generating false positives (14,15); and inconsistent testing of different variants on the same genes (16).

However, GWASs may be losing their status as a “paragon of open science” (17). GWASs have been criticized for being increasingly difficult to reproduce owing to inconsistent, un(der)reported methods and increasing use of private company data to enlarge samples. Commercial releases of partial data hinder replication. Finally, data come from unrepresentative samples (individuals who are predominantly affluent, of European ancestry, better educated, and/or healthy volunteers).

For science to remain open, existing research standards must be consistently implemented and constantly refined (1). Recent literature has addressed open science in psychiatry (18, 19, 20) or GWASs (21,22), but not specifically psychiatric GWASs. When reviews have addressed psychiatric genetics, they have prioritized replicability and reproducibility (23,24). These otherwise crucial dimensions are not the only facets of open scholarship. Here, we provide a primer on implementing open science practices in psychiatric genetics at multiple research steps (Figure 1): topic choice and authorship order, study design and preregistration, data access, open code, informative figures, inclusive language, reporting standards, preprints, nonacademic (citizen scientist) involvement, and research distribution. We identify common issues, suggest solutions, and include resources for every step.

Figure 1.

Open science practices at every research step, values which these actions support, and barriers to open science implementation. Figure created with Biorender.com.

Research Topic and Authorship

Several open science values–integrity, transparency, equity, and accessibility–are intertwined with study conceptualization.

Choice of Research Topic

Patient experts and researchers increasingly recognize that patients and community representatives should be successfully involved in every step of the research process (25). Such inclusion ensures that research addresses nonstigmatizing topics that are of the highest benefit to community-specific health needs (26).

Equity in Authorship and Author Order

Transparency and equity should be considered in authorship decisions. Authorship disclosures may be standardized forms of contributions, e.g., Contributor Roles Taxonomy (CRediT) (27). However, authorship is academic currency, but author order is still unequally attributed to men over women (28). Liboiron et al. (29) argued for equity as the guiding principle in establishing author order, considering care work and unequal division of labor. Care work (mentoring and teaching) is poorly acknowledged and disproportionately delegated to women and early-career or marginalized scholars (30). However, care work is vital to early-career scholars’ skillsets and well-being in research groups (31). Thus, we argue that care work must be credited, e.g., using CRediT (27).

To improve transparency and equity, we suggest that, prior to any project, researchers declare detailed criteria for determining author order (32, 33, 34), such as extent of contribution, author seniority, and the degree/rarity or number of biological samples contributed by each collaborator. Policies could define a significant contribution, e.g., whether author order is based on the number of completed tasks and how these tasks are quantified. This is crucial for studies that require different types of expertise.

Another solution is designating multiple corresponding authors for specific analyses, thereby distributing extra credit among authors who are responsible for technical contributions (e.g., programmers, statisticians). These authors are often placed in less desirable middle positions in author lists (35) or, historically in the case of female programmers, given acknowledgments but not authorship (36). Unfortunately, joint authorships are not consistently acknowledged (37,38). A novel approach proposes crediting authors throughout the manuscript text, thus making the contributions of each author clearer to readers, including hiring committees (39). While implementing this approach may prove complicated, we agree that changes in hiring practices are necessary. Diverse contributions should not be underappreciated if they do not result in first or senior authorship.

Equality in Citations

Genetic studies fail to represent global genetic diversity and come predominantly from Western sites (40), which have effectively monopolized the literature. Currently limited ways of redressing this imbalance are transparency about how a lack of global perspective impacts results and limits generalizability of study findings and referencing both large-scale, European ancestry-skewed studies and smaller studies of more diverse populations. Where studies are unavailable, researchers should cite theoretical work, protocols, and software by underrepresented scholars.

Study Design

Next, we briefly outline key study design considerations related to research clarity, reproducibility, replicability, and diversity.

Preventing Researcher Bias in Secondary Data Analysis

Secondary data analysis is common in psychiatric genetics. However, knowledge of data before analysis may introduce cognitive bias, thus distorting conclusions. Several solutions may limit bias (41): 1) a preregistration (Preregistration and Registered Reports); 2) sampling (e.g., hold out samples for exploratory research); 3) blinded analyses (42) or scrambling data (41); 4) multiverse analyses (testing whether findings remain consistent across multiple paradigms, e.g., frequentist and Bayesian analyses) (43); or 5) coordinated analyses (testing models across separate large-scale cohorts) (44).

Clear Phenotype Specification

Heterogeneous psychiatric phenotypes are frequently defined under a single diagnosis (45). Granular phenotypes (symptoms, brain structures, endophenotypes), which are difficult to obtain retrospectively, may be subject to stricter data sharing restrictions or may be difficult to harmonize across studies. Detailed publication of all phenotypic information, where appropriate, may partially address concerns about heterogeneity.

Another issue is phenotyping quality in genomic datasets that are linked to electronic health records. Electronic health records may reflect diagnostic difficulties (e.g., rare phenotypes), stigma, or race- or gender-specific bias in admission, diagnosis, and treatment. These biases may be ameliorated through 1) replication of phenotypic and genetic associations; 2) accounting for order, length, and time between episodes and treatment to avoid conflating unrelated episodes into a diagnosis; and 3) using known disorder biology to verify diagnoses, e.g., using polygenic scores or genetic correlations to disentangle whether diagnoses match their predicted underlying biology (46).

Studies should report what details were requested, whether questionnaires were applied across the cohort (e.g., whether trauma histories were obtained from posttraumatic stress disorder controls or only cases), who determined the phenotypes, and potential inaccuracies. It is also crucial to state whether results were compared across samples with different phenotype operationalizations. For instance, a GWAS of age of onset in psychiatric and nonpsychiatric disorders yielded different significant hits according to the age of onset definitions that were applied (47). Finally, data collection protocols should emphasize the maximum level of detail so that analysts can eventually understand sources of variability in the data (48).

Sample Size Justifications

Sample size justifications should include power calculations or clarification of why power calculations are absent (a whole population study; resource constraints; heuristics based on literature or norms in the field; or no reason to specify power, e.g., no clear inference goal) (49).

Ameliorating Eurocentric Bias

In 2017, 88% of GWAS findings came from European ancestry samples. Indigenous samples represented only 0.02% of GWAS findings (40). Genetic association statistics are not generalizable across ancestries (50). Eurocentric bias is also present in pharmacogenetics. While 30% of studies use European ancestry samples, ancestry reporting is frequently incomplete or vague. This results in biased assessments of drug targets, which are potentially not relevant across populations (51).

To ameliorate Eurocentric bias, researchers should analyze data from minoritized populations. Data from different ancestries should be analyzed separately. Researchers should acknowledge underpowered samples (52) or use well-powered international biobank data or summary statistics from biobank networks (8). Researchers should also implement pipelines for multiancestral data analysis (53, 54, 55, 56).

Preregistration and Registered Reports

Preregistration specifies hypotheses (or research questions for hypothesis-free designs), study design, and analysis plans before data collection or secondary data analyses begin (57). Registered reports (RRs) are peer-reviewed, preregistered methods and data collection protocols (58) that result in publication regardless of study outcome. We encourage journals to publish RRs, null results, and nonreplications. Guidance on preregistrations/RRs, including in consortia, is detailed elsewhere (57,59, 60, 61, 62, 63).

Despite increasing guidance, not all aspects of preregistration have been resolved. Preregistrations require substantial effort, but they can be completed using research plans required for proposals to ethics boards (64) or consortia for data access (59). Another issue is the cost of open RR publications, which is paid early in the process. This may prevent researchers from choosing RRs. We call on journals to provide clear RR pricing guidance and on funders to provide incentives (e.g., https://www.cancerresearchuk.org/funding-for-researchers/how-we-deliver-research/positive-research-culture/registered-reports).

Data Access and Data Sharing

Open annotated data and metadata are the gold standards for replicability. However, reasonable privacy concerns prevent making much genetic data fully open. The European Commission Directorate-General for Research and Innovation has called for data “as open as possible and as closed as necessary”: “open” where possible for reuse, but “closed” to protect privacy (65). Studies of public perception of genomic data sharing in 22 countries demonstrated that attitudes vary on increasing public trust in the process (66) and willingness to share data with for-profit researchers and medical doctors (67) in return for results (68). Thus, we argue that participant safety and wishes must be prioritized and should not be assumed to be universal.

Nonetheless, respect for study participants is at odds with full data access. The onus falls on researchers to optimize approaches for making data accessible without risking identification of participants or to develop methods whereby summary statistics (which aggregate cohort-level genetic information) are sufficient to perform analyses. For multiancestral projects, ancestry-specific summary statistics are also necessary (69).

Within Consortia

The “secondary analysis proposal” model gives access to deidentified individual-level genotype and phenotype data to researchers with an approved analysis plan [implemented, for example, by the PGC (70)]. Where external users cannot easily access data, sites may employ a confederate model whereby analytical procedures and scripts are shared or site analysts run analyses and share summary data.

Within Institutions

Data sharing between teams may accelerate collaboration, reduce time spent on quality control (Data Quality Control and Analysis), and enable researchers to detect errors prior to publication. Institutions should incentivize such data sharing. A blueprint is the Quality-Ethics-Open Science-Translation Center (Berlin Institute of Health, Germany), which funds institution-wide data sharing (71).

A novel approach to data sharing is adversarial collaborations. Adversarial collaborations have research teams jointly access the data but test opposing hypotheses. Teams also reproduce analyses of their opponents and edit each other’s descriptions of theory to prevent misrepresentation of opposing views (72).

Summary-Level Data

Alternatively, researchers may prioritize analyzing open summary statistics, including associated statistics such as linkage disequilibrium matrices. Gold standard approaches include, e.g., GCTA-COJO (genome-wide complex trait analysis-conditional and joint analysis) (73) for conditional analysis of GWAS summary statistics; FINEMAP (74) and PAINTOR (75) for fine-mapping loci; COLOC (76, 77, 78) and summary data-based Mendelian randomization (79) for expression quantitative trait loci annotation of GWAS summary statistics; and S-PrediXcan (80) for transcriptomic imputation. However, we note that these approaches perform best with linkage disequilibrium matrices of the original data.

Another issue may be limited access to summary-level data from commercial entities (17). We believe that journals could enforce data sharing, e.g., by accepting a publication only after its authors have provided complete summary statistics.

Indigenous Data Sovereignty

Indigenous data sovereignty (IDS) describes the right of Indigenous peoples to control access to data and samples that are collected on their lands and from their communities (81,82). Although IDS precludes broad data access, the benefits to open science are nevertheless myriad: inclusive research development, innovation, and improved citizen engagement (because Indigenous communities fear that fully open data could result in misinterpretation of eventual results and increased stigma) (81,82). Increased IDS will enhance accountability through transparency of data and sample handling and improve equity when Indigenous communities benefit through publication, research, or commercialization. Increased Indigenous participation would increase diversity and inclusion in genetics (81). These values, together with degrees of improved transparency and equity, are also open science values.

IDS guidance is summarized in the CARE Principles for Indigenous Data Governance (collective benefit, authority to control, responsibility and ethics) (83). CARE principles reflect tribal expectations of research conduct: tribal consent and input are followed throughout, e.g., when and how samples are returned to honor participants’ spiritual practices; research benefits reflect tribal priorities; Indigenous individuals are employed in research projects and compensated; research materials and data belong to tribes who govern access, review research progress, provide institutional review boards, and grant secondary uses of data and research materials; findings are returned to communities; researchers respect Indigenous cultural and spiritual values; researchers and Indigenous communities are equal collaborators (84).

Data Quality Control and Analysis

Quality Control

Genetic data quality remains inconsistent (17). We suggest consistently following and describing completed quality control and imputation protocols and explaining any deviations from protocols in manuscripts and open datasets (which frequently lack information on the quality control protocol that was applied).

Open Source

Open source tools are commonly used and potentially improve replicability and reproducibility [e.g., PLINK (22,85)]. However, while software used for analysis is often shared, software and scripts for preprocessing data are not, which limits reproducibility (86).

Detailed Record of Analytical Steps and Open Code

While preregistrations (Preregistration and RRs) ideally provide specific research plans, these plans can expand (e.g., following reviewer suggestions). Researchers should keep detailed, transparent analysis records, e.g., in open online laboratory notebooks (87). Code should be openly deposited with version control and a codebook in which all variables are summarized (88). Workflows should be automated (files are not edited manually but result from how the script executes), with labeled input and output files. Relationships between data, data preprocessing, software, and analysis outcomes should be explained in a compendium article (89), e.g., an interactive website Jupyter notebook, or R Markdown files, which are convertible into books, articles, or websites (90).

Bioinformatics Tools

Development and maintenance of bioinformatics tools provides research tools and hands-on software development training, which most biologists (or psychiatry researchers) are not trained in. However, tool development is unscalable without continuous funding for maintenance, reproducibility, and training. Tool development is also not credited with coveted senior authorship, and software is not consistently cited (91). We echo the call from the bioinformatics community for consistent funding and counting software development toward career progression. To that end, researchers could make their bioinformatic work citable with persistent handles (e.g., generated through Zenodo, https://zenodo.org/) and consistently attribute authorship of software, data visualization tools, etc.

Reporting

Language

Detailed reporting supports transparency, replicability, and reproducibility but may also be in tension with open science values. Using language that participants prefer (to describe their experience, communities, etc.) improves inclusivity and reproduces information shared by participants. However, using unstandardized descriptors contradicts facilitating reproducibility through shared terminology. Consequently, we suggest that researchers provide descriptions from participants, operationalizations (e.g., diagnostic criteria), or, if these are unavailable, inclusion/exclusion criteria (e.g., self-diagnosis, disorder for which formal diagnostic criteria are lacking, symptom types, frequency, intensity, etc.). Below, we also briefly outline discussions about language choices for commonly reported data as a starting point for researchers.

Ancestry, Race, and Ethnicity

The use of race as interchangeable with ancestry and ethnicity is decreasing (92) but has not stopped. The terms have different meanings. Race is a sociopolitical concept whereby group identification is based on often stereotypical or politically influenced defining physical characteristics, e.g., skin pigmentation (54,93). Ethnicity, on the other hand, is a category that is based on group belonging through shared language and traditions. Ethnic groups may share genetic factors because of similar ancestral origins or be self-identified based on shared culture (54,94). Genetic ancestry refers to populations from which an individual has descended, which is reflected in DNA inherited from recent biological ancestors (54). Biogeographical ancestry labeling (African, Asian, and European) has increased (92). However, it has been criticized for being reminiscent of historical racial taxonomies (95), not capturing heterogeneity within local subpopulations (96), and being inconsistent with global genetic variation, which is continuous, not discrete continental (93). In addition, individuals who are not monoracial are frequently grouped into monolithic, imprecise, or othering categories, e.g., multicultural, admixed, bi/multiracial, and other (97). These descriptors are unacceptable.

At a minimum, researchers should follow antiracist genetics publication recommendations (94), which require clearly stated definitions and reasons for addressing race and ancestry. Race impacts health care delivery, disorder etiologies, and health outcomes. Accounting for race/ethnicity in modeling is necessary, but explicitly as a marker of inequality, never as a proxy for genetic ancestry (54). Ideally, researchers should work in interdisciplinary teams with affected communities (93) and consider outlining their own identities and motivations in manuscripts to build trust with affected communities (97).

Sex and Gender

Sex and gender are not synonymous, essentialist constructs. Gender encompasses sociopolitically constructed roles, behaviors, and identities (98). Sex is a set of physical and physiological variables (genitalia, gametes, karyotypes, gene expression, hormones) (98) that do not necessarily or fully determine sex (99). Common issues with reporting gender/sex differences include inconsistent or conflated use of sex and gender; sex or sample sizes per sex not reported; statistical evidence for claimed gender differences not provided; and no stated criteria for determining or ascertaining sex/gender (100,101).

Diagnosis and Patient Identity

Two general ways of describing patients are person-first language (e.g., a person with autism) and identity-first language (e.g., an autistic person). Person-first language prioritizes an individual, while communities, e.g., the English-speaking autism community, argue that identity-first descriptions emphasize that diagnoses can be proudly integral to one’s identity (102). Participant preference should be respected; where unknown, researchers should be sensitive to historical agendas, priorities, and different experiences within the community (103). We also suggest that researchers transparently report how descriptions were determined.

Reporting Guidelines

Health care studies insufficiently report statistical methods, missing values, or reporting guidelines followed (104). Multiple reporting standards exist for common genetic designs (105, 106, 107, 108). Furthermore, researchers should ensure that their results are not overinterpreted. Researchers should state why the evidence may be causal and clearly communicate any uncertainty (109, 110, 111), including in press releases.

Figures

Good figures clearly communicate approach, results, and key messages, thus improving transparency and reproducibility. To make figures accessible, researchers should address font size, readability, and color selection for individuals with visual impairments or color blindness. Accessibility and inclusivity also extend to imagery, mindful of lived experience. For example, eating disorder studies should avoid potentially triggering visuals, e.g., scales or distorted body images in mirrors. Graphics for case-control groups should not depict a singular gender presentation/race/body type as a sample case. Differences between cases and controls could be labeled with neutral colors (not green and red, implying good and bad for controls and cases; these colors are also unreadable to individuals with color-vision deficiencies). We provide free resources for creating accessible figures in Table S1.

Crucially, researchers should be aware of how data visualizations may be misinterpreted through a racist lens and co-opted by far-right extremists to further white supremacy, especially given that much of the genetic research is Eurocentric (Study Design). Researchers have to anticipate how their visualizations, out of context and without labels, could be misconstrued. A preventive solution would be to standardize plot presentation in the field (112).

Preprints

Preprints, nonpeer reviewed papers, are now integral to fast and free research sharing (113). Preprints also streamline submission to journals. In bioRxiv, 139 journals relevant to psychiatric genetics, psychiatry, and bioinformatics endorsed direct journal transfer. However, the preprint explosion during the COVID-19 pandemic highlighted limitations of rapid sharing, which we suggest that researchers be mindful of. Rapidly generated preprints, some of which may be methodologically flawed or fraudulent, may end up being integrated into unreliable meta-analyses or flawed health policy. Furthermore, flawed studies may lead to wasteful or harmful follow-up research even after retraction of the original preprints (114).

Citizen Science and Participant Inclusion

Knowledge coproduction (between patient experts and researchers) may be difficult due to pressures of timescales and funding (115). More extremely, coproduction could leave patients feeling tokenized or invalidated when their experience is deemed wrong against the views of psychiatrists or Eurocentric models of knowledge (116). Specific approaches may foster inclusive collaborations (115,116) including the following: 1) resources dedicated to supporting relationships with patient experts during and between projects; 2) open events where patients and researchers explore a specific key issue to identify creative solutions; 3) published evidence of the impact of coproduced research, supporting continuous collaborations; and 4) careful consultations between researchers and patient experts. Researchers should prepare to address boundaries to participation, conflicting views, and difficult journeys through health services (e.g., due to racism or coercion) (116,117). Patient experts should not be expected to share sensitive community knowledge. It should be clear how researchers will use knowledge that is obtained from patient experts (118).

In citizen science, nonacademic members of the public are active research collaborators, not passive data donors. Collaborations should reflect citizen science values: autonomy, fun, respect, altruism, inclusivity, openness, reciprocity, and solidarity (119,120). Researchers should plan how to navigate divergent collaborator priorities, e.g., when and how to release results to comply with institutional or regulatory requirements (121). In terms of compensation, citizen scientists wish for accessible data, clearly communicated findings, and acknowledgments in publications (122). Credit or intellectual property compensation, to be legally binding, needs to be detailed in contracts or institutional policy (121).

Research Distribution

Researchers should consider how to communicate findings in such a way as to uphold open science values. While psychiatric geneticists support returning results to patients, most do not deem their knowledge of the process to be adequate (123) [although see recent international guidelines (124, 125, 126)]. In addition, scientists incorrectly assume that publication in peer-reviewed journals makes results accessible (121). Scientific publications may be publicly unavailable or written in a language that is difficult for nonspecialists to understand. To ensure access, American genomic citizen science projects provide links to open access publications on project websites (127).

Writing manuscripts in a nontechnical manner would be a good general practice. However, we acknowledge that it can be challenging. Extra (technical) details of analyses are required for replicability/reproducibility, but they decrease the overall accessibility of manuscripts. Several solutions could be put in place; for example, manuscripts could be accompanied by nontechnical press releases sent to participants in automated e-mails or letters. Researchers already explain their work or summarize talks (128) using blogs or social media posts (129). Accessible materials explaining research to nonacademic readers could be provided, e.g., on manuscript companion websites and websites of research societies (e.g., https://ispg.net/resources/educational-presentations, https://pgc.unc.edu/for-the-public/basic-genetics/), laboratories, or patient advocacy groups.

An alternative to written communication is live events. Accessibility practices include live captions/transcriptions, inclusive language, and translations, including sign language; affordable online conferences and ensuring that scholars who require visas can attend meetings; even division of time among speakers; and implementing a code of conduct (130). Alternatively, nonreal-time web conferences allow attendees to watch presentations, while remaining friendly toward time zones and caretaking commitments. Nonlive conferences also do not require a stable Internet connection (131).

Finally, because psychiatric genetics requires multidisciplinary technical training, open teaching materials would support learning about the newest techniques. Platforms such as Open Science Framework (https://osf.io/) or Zenodo permit free content uploads (slides, posters, and recordings of talks, which should include captions, transcripts, and translations). These teaching materials can have citable digital object identifiers generated. Textbooks can also be easily uploaded online from R Markdown files (Data Quality Control and Analysis).

Conclusions

Methodological rigor and careful collection of open data and their transparent, easy-to-follow presentation are key to improving the reproducibility, replicability, and accessibility of psychiatric genetics studies. However, psychiatric genetics remaining at the forefront of open science will depend on the ongoing support of and advocacy for collaborations and credit for junior, minority, technical, and nonprofessional researchers. Continuous funding is needed for research with non-European ancestry samples in a manner which respects the rights and privacy of underrepresented groups, following the increasingly embraced call to make data “as open as possible, as closed as necessary.” Funding is also required for the continuous software development that is necessary to analyze these complex data. Finally, consistent funding and institutional change of research norms would support researchers by providing them with extra time needed to run reproducible and replicable studies, providing clear research explanations for the public, creating open teaching materials, and maintaining ongoing collaborations with patient experts.