Reproducibility in chemistry research

Abstract

Chemistry is a reproducible science whose pillars - synthesis and analysis - actually comprise a huge collection of highly reproducible experimental methods to synthesize and analyze substances. The historical development of chemistry, furthermore, shows that reproducibility of methods has been the companion of novelty and creative innovation. The “publish or perish” principle dominating global academia since over two decades, however, intrinsically contributes to the publication of non-reproducible research outcomes also in chemistry. A study on reproducibility of chemistry research seems therefore timely, especially now that chemists are slowly but inevitably adopting open science and its tools such as the preprint, open access, and data sharing. We conclude presenting three simple guidelines for enhanced publication of research findings in chemistry.

1. Introduction

The words “reproducibility” and “replicability” applied to the results of scientific work are often used as synonyms to indicate the ability of reproducing or replicating prior work [1]. According to a committee appointed in the late 2010s by the National Academies of Sciences of the United States of America, reproducibility would be “obtaining consistent results using the same input data; computational steps, methods, and code; and conditions of analysis” [2], whereas replicability is “obtaining consistent results across studies aimed at answering the same scientific question, each of which has obtained its own data … given the level of uncertainty inherent in the system under study” [2]. Drummond, in his turn, has suggested that replicability would “the impoverished version” [3] of reproducibility and its only benefit would be as a policing tool, preventing scientific fraud. Noting that scientific studies have intrinsic irreducible uncertainties (due to random processes in the system under study or in the ability to control that system, or limitations in the precision of measurement), the aforementioned committee concluded that researchers should include “a clear, specific, and complete description of how the reported result was reached” [2]; while academic institutions should develop “training of researchers in the proper use of statistical analysis and inference” [2].

Since the early 2000s a number of studies reported poor replicability in disciplines such as biomedicine [[4], [5], [6]] as well as in social sciences. For example, in psychology attempts to replicate 100 studies published in three different journals failed to validate the hypothesis of the original study in 64 cases out of 100 [7].

Chemistry, in general, is a highly reproducible science. Its practice chiefly concerns the synthesis and the analysis of new or known substances [8]. Chemical synthesis and chemical analysis, indeed, comprise a huge collection of highly reproducible experimental methods. If synthetic or analytical methods were not reproducible, the world's largest manufacturing industry (the chemical industry) and the analytical chemistry industrial sector with multiple manufacturers of analytical instruments and “analytical-grade” chemical reagents would not exist.

For example, since decades chemical companies manufacturing solid catalysts sell said catalysts of guaranteed, reproducible performance under specified experimental conditions. From jet's fuel through ammonia and advanced polymers, the chemical industry uses said commercial catalysts to reliably manufacture nearly all its products.

Little scholarly research has been published to date on reproducibility in chemistry. For example, a Boolean search carried out in late 2023 on a research database using the “reproducibility in chemistry” query returned only nine results [9]. Yet, in 2016 a scientific journal conducted an online survey of 1576 scientists in six disciplines (chemistry, physics and engineering, biology, earth and environment, medicine, other), 106 of whom were chemists. Nearly 90 % of them claimed to have failed to replicate the results of other colleagues [10]. Strangely enough, 63 % of the same chemist sample could not reproduce their own work. Based also on this poll's outcomes, in 2019 the American Chemical Society, whose publishing branch (ACS Publishing) is a large and reputable chemistry publisher, published a policy for presenting reproducible biological chemistry data [11].

Three years before, the editor of Organic Syntheses reported that in years comprised between 2010 and 2016 7.5 % of the submissions were denied publication because the yield or the selectivity reported could not be reproduced “within a reasonable range” [12] in the laboratory of one of the journal's editorial board members. Organic chemistry reactions seeking publication in Organic Syntheses, indeed, must be reproduced twice in the laboratory of a member of the journal's board. The 2010–2016 result, though, was an improvement over the 1982–2005 period when rejection rate for the same reason was about 12 % [13].

Research chemists publish a large (and growing) annual output of scientific papers (176,000 papers in 2017, excluding papers in materials chemistry) [14]. Between the 2008–2012 and 2013–2017 the number of papers indexed in the research field of “Chemistry” by a large research database went from about 725,000 to nearly 880,000 [14]. To host such large and increasing number of articles, since the early 2000s scientific publishers have “launched” numerous new chemistry journals. For example, in 2022 the scientific database Scimago ranked in the subject area “Chemistry” (all subject categories, all countries) 916 journals (of which 253 were OA journals) [15]. Ten years earlier, in 2012, the journals ranked by the same database were 812 (of which 157 OA venues) [15].

Scientific publishers compete for manuscript submissions, while research chemists “strive to publish and not perish” [16]. The latter pressure incentivizes publication of poorly reproducible research also in chemistry, beyond (see below) the intrinsic complexity of certain subfields of chemistry research where tiny variations in the experimental conditions may lead to dramatic changes in the reported results. As recently put it by the editors of one such new journals established by the ACS in 2011, “the demands on researchers to advance quickly in their work, to make broad claims of imminent potential impact, and to popularize their results …, have affected how science is conducted and communicated. It is hardly surprising, then, that concerns about reproducibility have re-emerged … in many disciplines, including catalysis [17]”.

A study on reproducibility of chemistry research seems therefore timely, especially now that chemists are slowly but inevitably uptaking open science principles and tools, such as the preprint [18], enabling open access (OA) to published studies, and data sharing [19].

2. Results and discussion

In the following, we first identify the main findings concerning reproducibility studies in three selected subfields of chemistry research (materials and supramolecular chemistry, electroorganic synthesis and catalysis). We then summarize selected key lessons learned from these and related studies including other spanning from Liebig's kaliapparat through NMR and computational chemistry investigations.

2.1. Materials and supramolecular chemistry

In 2016 Broom and Hirscher summarized results showing evidence of poor reproducibility of data concerning hydrogen adsorption measurements in porous materials (investigated for H₂ storage) published in the previous two decades (1996–2015) [20]. The team found that results on H₂ sorption on carbon nanotubes, metal-organic frameworks (MOFs), carbon nanofibres, conducting polymers (polyaniline and polypyrrole) and boron nitride nanotubes were systematically overestimated. Most of the irreproducible results were of methodological nature, and due to contamination of materials investigated. The scholars concluded that “more, as yet unidentified, irreproducible data is likely to exist in the current literature” [20].

Indeed, the subsequent year Sholl and co-workers reported similar results regarding CO₂ adsorption measurements in MOFs [21]. Carrying out a meta-analysis of CO₂ adsorption isotherms from a research database including data from peer reviewed studies, they found that ∼20 % of isotherms could be classified as outliers according to Tukey's boxplot method for data analysis [22].

Getting to the chemical reactivity of functional materials, Thielemans and Labet in 2011 were among the first to emphasize the need to improve the reproducibility of results [23]. The same reactions performed on cellulose nanocrystals derived from different hydrolysis samples gave different results. They ascribed this outcome to the presence of molecules produced during the hydrolysis at the surface of different nanocellulose samples, and identified Soxhlet extraction with EtOH before reaction as the most effective method to remove the adsorbed species [23]. A few months later, Meijer and co-workers reported that the spectroscopic data of the reaction product of the polymerization of the chiral oligo(para-phenylenevinylene) (S)-OPV were reproducible only when the polymer synthesis protocol was strictly followed [24]. The latter polymerization indeed relies on weak interactions such as hydrogen-bonded dimer formation that assemble into helical stacks in alkanes [24]. Showing a number of issues affecting reproducibility of supramolecular systems synthetic routes, the same researchers suggested authors to carefully report also negative results [25].

Broom and Hirscher recently suggested six key stages of characterising the H₂ storage properties of materials and addressing problematic data (sample preparation, instrument set-up and testing, troubleshooting, measurement, data processing, assessing results), alongside twelve reporting guidelines [26]. Similarly, Scholl and co-workers lately published three recommendations aimed at researchers measuring the properties of “nonbiological” materials [27].

Reporting that in papers including the outcomes of computer calculations applied to materials results are often non-reproducible because the authors do not report key data used for analysis (such as molecular and crystal structures, and the input files for calculations and the software), Coudert recommended to open the data [28]. This should be done starting from structures that should be provided as structure files, publishing the input and output files and preferably use software that is either open source or freely available [28]. Similarly, Pauli and co-workers in 2016 recommended authors to always publish the original NMR data to enhance the reproducibility as well as the traceability of results [29]. The team indeed found that that in NMR spectroscopy studies aimed at structural elucidation and quantification of natural products, poor reproducibility is often due to the lack in availability of original data [29].

2.2. Electroorganic synthesis

Electroorganic synthesis requires a scientific background including advanced knowledge of electrochemistry. In 2021Waldovagel and co-workers summarized the “crucial, but under-explored parameters” [30] affecting electroorganic syntheses in constant-current electrolysis cells: electrode geometry and surface features, membrane, electrode tilt angle, stirring, and the distance between the electrodes. Variation in the parameters and methods reported in the experiments directly affects the reproducibility of the results [30]. For example, not only the composition, but also the surface area and 3D geometry of the working electrode affect organic electrosynthesis. It is therefore important to precisely report the type of graphite employed as carbon materials commonly used as anode materials, because different forms of graphite have different porosity and surface structure. For example, the selective mesylation of naphthalene at the 1-position achieves the highest yield when using isostatic graphite [31], with only a carbon-felt electrode approaching the best yield of 56 % found with isostatic graphite.

2.3. Catalysis

As mentioned above, catalyst manufacturers sell to customers in the chemical industry catalysts of guaranteed, reproducible performance under specified experimental conditions. Using these catalysts under said conditions, the chemical industry reliably manufactures nearly all its products from “bulk” chemicals such as ammonia and hydrogen peroxide through “fine” chemicals such as the organic light emitting diodes (OLEDs) to be used in state-of-the-art mobile phone screens. To further improve the quality of data in catalysis research in Germany, the country that hosts the world's largest catalyst manufacturer, academic and industrial research chemists recently launched a consortium aiming to make available to industrial and academic researchers in Germany large amounts of high-quality data from catalysis research [32].

Photocatalysis, for instance, is a subfield of catalysis research and technology that has been greatly affected by non-reproducible data due to the obsolescence of lighting technology used for decades, the use of batch reactors with limited light transmission to reactants, and poor reporting of results. On the other hand, today's high-quality light emitting diode (LED) sources have replaced the mercury vapor lamps previously used in photocatalysis, with dramatic reduction in the amount of heat transferred to the reactor by inefficient lighting technology. Besides, and even more important in enhancing reproducibility, was the introduction of flow photoreactors in place of batch reactors for photochemical synthesis (see below).

To enable reproducibility in heterogeneous photocatalysis, Takanabe and Qureshi in 2017 suggested to report the outcomes of heterogeneously catalyzed photocatalytic reactions including i) reactant conversion or product evolution rates; ii) the photon flux (wavelength, type of lamp, use of filters etc.); iii) partial pressures of reactants and sacrificial reagents; iv) nature and amount of the solution employed, including electrolyte concentration and pH; amount of photocatalyst and co-catalyst; flow rate (if reaction is carried out under flow) or reactor volume (if reaction is carried out in batch); v) dimension and photograph of the reactor employed [33].

Similarly, today's research in catalysis has long identified the key catalytic role of hyper-reactive metal clusters and metal atoms leached in solution by reactants contaminated with tiny amounts of metal impurities. For example, a level of 50 ppb of Pd contaminating a batch of commercial Na₂CO₃ is enough to mediate formation of biaryl products in the Suzuki-Miyaura reaction between aryl halides and boronic acid [34]. Similarly, all 60 magnetic stir bars previously used in catalytic syntheses were found to be contaminated with metal nanoparticles or microparticles, including palladium, gold, platinum, cobalt, chromium and iron [35]. The amount of Pd leached from used stir bars is sufficient to catalyze the same Suzuki-Miyaura cross-coupling reaction carried out in methanol at 50 °C, in one case even matching the 17 % yield observed when using a commercial Pd/C catalyst.

2.4. Lessons learned

The main lesson learned from the present investigation is that in chemistry research there is no conflict between novelty and reproducibility, as feared by Drummond for other scientific disciplines such as modeling and data science [36]. For example, the fact that many heterogeneously catalyzed photocatalytic reactions were experimentally cumbersome and poorly reproducible eventually led research chemists to address the key reason explaining also the poor industrial uptake of photocatalytic syntheses in the chemical industry, namely the batch photoreactor in which most light is absorbed by the solution layer within 1 mm from the lamp. In 2005 thus Booker-Milburn and co-workers resolved this issue that limited the industrial uptake of photochemistry for nearly a century by inventing the first compact flow reactor comprised of transparent fluoropolymer tube wrapped around the light source [37]. Today, using vastly improved LED lighting sources and related transparent polymers, the same flow photoreactor is used by the fine chemical industry for the photocatalytic synthesis of many valued chemical products eventually using photons as reactants in an effective and efficient way [38].

While in 2016 Broom and Hirscher reported poor reproducibility of data concerning H₂ adsorption measurements in porous materials [20], in 2005 Usselman, Reinhart and Foulser reported the successful outcomes of reproducing Liebig's analyses of racemic (tartaric) acid, cinchonine, narcotine, and urea first carried out in 1831 in Germany using a self-made Liebig's kaliapparat [39] (or potash apparatus: a triangular, hollow tubular glass device, with five bulbs inserted along its three arms), following the procedures published by Liebig in 1837 [40]. In detail, experiments could be successfully replicated for the elemental analysis of H and C, but not for N. Indeed, especially for nitrogen-poor substances, air's nitrogen was interfering in the original kaliapparat (requiring a subsequent innovation from Dumas in 1833 who added lead carbonate at the exit of the combustion tube). The kaliapparat traps the effluent H₂O and CO₂ gases respectively as hexahydrate salt of anhydrous CaCl₂ placed in one bulb and as potassium hydrogencarbonate in three bulbs filled with concentrated potash solution. Using the kaliapparat, Liebig was able to assess the amount of hydrogen, carbon and nitrogen present in the burned compound (elemental analysis) and to assign atomistic formulas. The team reconstructed Liebig's kaliapparat and obtained accurate results for carbon and hydrogen content, but inaccurate results for nitrogen (likely due to today's regulations which forbid manual handling of toxic mercury) [39].

In other words, extending the analysis to the 19th century, one may even learn that along with ease of use it was exactly the high reproducibility of results obtained with Liebig's kaliapparat also by beginner students that allowed Liebig “to transform chemistry's laboratory economy, developing a system of large scale practical training” [41], allowing Liebig “to do research in Giessen, Germany - what was then a backwater of the science” [41].

“It's mind-boggling”, Rocke wrote commenting the results of reproducing in 2005 the analyses of organic compounds using Liebig's kaliapparat dating back to 1831, “that the simple apparatus can give you the quality of modern analyses” [42]. On the other hand, stability and lifetime data for perovskite and dye-sensitized solar cells reported in 2015–2016 in 261 aging tests published in peer-reviewed studies are vastly unreliable [43]. The aging data indeed were typically reported only for one cell, and the basic parameters (temperature, humidity, and intensity of both UV and visible light) were given only in one-third of the 261 reviewed aging studies.

As mentioned above, the historical development of chemistry shows evidence that reproducibility has been the companion (and not the enemy [36]) of novelty and creative innovation.

3. Conclusions

Reviewing the state of reproducibility in three important areas of today's chemistry research (materials and supramolecular chemistry, electroorganic synthesis, and catalysis) shows that research chemists are aware of the need to improve reproducibility, particularly in certain fields, and have identified technical means to do it. Chemistry and material science journal's editors are also aware of the need to improve reproducibility. For example, ChemCatChem (a catalysis journal) requires that “the structure and composition of all compounds and materials central to the manuscript must be disclosed in the main text or in the Supporting Information” [44].

In chemistry too, however, the “publish or perish” principle contributes to the publication of non-reproducible results. The latter publication pressure since over two decades dominating global academia [16], requires researchers to publish in scholarly journals rapidly and continually for obtaining and renewing research funding, as well as for promotion (faculty position), and university ranking. From the aforementioned recommendations to develop educational programs of researchers in the use of statistical analysis [2], through the open science principle to publicly share data so as to make them easily findable, accessible, interoperable, and reusable [19], there are several technical means to improve reproducibility in chemistry research.

At a more fundamental level, aware that poor reproducibility actually originates from the “publish or perish” pressure [16], research chemists may wish to consider the research reproducibility issue from the standpoint of intellectual humility. Chemists too need to rediscover the need to write and publish “intellectually humble research articles” [45], namely scientific studies in which “the limitations of work are owned and their consequences explicitly incorporated into the conclusions” [45]. Included in these limitations are reproducibility issues. Were experiments repeated affording consistent results? Again, seen from this standpoint, the issue of reproducibility becomes an opportunity to improve the overall quality of published research.

Investigating hyping items (from a list containing 400 hyping terms) in 100 documents submitted by British universities to government for research evaluation in 2014, Hyland and Jiang recently found that research chemists are the most hypers [46]. Among scholars in all investigated disciplines, chemists used on average 2.54 hyping terms every 100 words, significantly greater than researchers in the other fields. Taking into account said findings and willing to write and publish better and intellectually humble research articles, research chemists will abstain from using hyping terms and bold claims. In their place, chemists will insert in their manuscript videos and photographs showing key aspects of the newly developed process/products described in their study.

Table 1 presents three simple guidelines for enhanced publication of research findings in chemistry.

Table 1.

Guidelines for enhanced publication of research findings in chemistry.

Aim	How to achieve it
Publish intellectually humble research articles	Own the limitations of work and incorporate their consequences in the conclusions
Publish reproducible research findings	Carefully check the reproducibility of experimental procedures; publish videos and photographs
Publish articles free of hype	Avoid hyping terms and bold claims

Online publication of videos and photographs is a possibility offered by the internet that so far has been barely used when publishing academic research papers, even though including photographs in Supporting Information is now fairly common, at least in some areas of chemistry, and is even required and used extensively in some journals (e.g., Organic Syntheses). Publishers of journals in chemistry and all its numerous domains should encourage authors to publish manuscripts embedding videos and photographs of their experimental work. Then the publisher's journal production department should make said videos and photographs “easy to engage with as the figures in the text of their articles” [47] as it happens in leading life science journals such as eLife.

In chemistry too, the online version of a research article may be now considered its primary version [48], whereas the static version on paper becomes an impoverished version of the online study. For instance, when in 2023 Della Pina and Ciriminna wanted to show the ease and highly reproducible nature of the new process to entrap water-soluble graphene oxide molecules within the lattice of palladium nanoparticles, they published a video on the University of Milan's website, and inserted a link to the video on the preprint posted on ChemRxiv reporting the discovery [49]. Similarly, many other research chemists actively use videos to illustrate and publish the outcomes of their research activities. The Journal of Visualized Experiments cross-disciplinary journal, for example, by mid June 2024 had published 635 videos in the chemistry research section [50].

In conclusion, a reflection on the issue of reproducibility in today's chemistry research becomes an opportunity to improve the overall process of scholarly communication starting from its cornerstone, the scientific article. Following a number of studies aimed at improving scientific publishing in the open science and digital era [45,51,52], this study suggests further avenues on how to achieve said progress in chemistry and related disciplines.

Data availability

No new data have been produced for this study.

CRediT authorship contribution statement

Rosaria Ciriminna: Writing – review & editing, Methodology, Funding acquisition, Conceptualization. Giuseppe Angellotti: Investigation, Formal analysis. Giovanna Li Petri: Investigation, Formal analysis. Mario Pagliaro: Writing – original draft, Supervision, Methodology, Conceptualization.

Declaration of competing interest

The authors declare that they have no known competing financial interests or personal relationships that could have appeared to influence the work reported in this paper.