Believable Statements of Uncertainty and Believable Science

Abstract

Nearly fifty years ago, two landmark papers appeared that should have cured the problem of ambiguous uncertainty statements in published data. Eisenhart’s paper in Science called for statistically meaningful numbers, and Currie’s Analytical Chemistry paper revealed the wide range in common definitions of detection limit. Confusion and worse can result when uncertainties are misinterpreted or ignored. The recent stories of cold fusion, variable radioactive decay, and piezonuclear reactions provide cautionary examples in which prior probability has been neglected. We show examples from our laboratory and others to illustrate the fact that uncertainty depends on both statistical and scientific judgment.

Introduction

Two papers were published in 1968 that clearly set out the terms of discussion for two fundamental concepts in measurement science. Churchill Eisenhart’s four-page paper in Science, “Expression of the Uncertainties of Final Results” [1], pointed out the several meanings that the literature might be understood to imply by an uncertainty expressed in the shorthand form a ± b. “If no explanation is given, many persons will take ±b to signify bounds to the inaccuracy of a. Others may assume that b is the ‘standard error,’ or the ‘probable error,’ of a, and hence the uncertainty of a is at least ±3b, or ±4b, respectively. Still others may take b to be an indication merely of the imprecision of the individual measurements, that is, to be the ‘standard deviation,’ or the ‘average deviation,’ or the ‘probable error’ of a single observation.” As a consequence, far too great a fraction of the data in the scientific literature “cannot be critically evaluated because the minimum of essential information is not present.” Eisenhart recommended unambiguous and statistically valid procedures for expressing uncertainty which should have cleared the air and set the standards for future publications.

Also in 1968, Lloyd Currie’s paper in Analytical Chemistry, “Limits for Qualitative Detection and Quantitative Determination” [2], applied eight common definitions of detection limit from the literature to a simple measurement example in radiation counting, and showed that the resulting estimates cover nearly three orders of magnitude. He then re-examined from the statistical point of view of hypothesis testing what detection and measurement should mean in analytical chemistry, rigorously defining three quantities: critical level, detection limit, and determination limit. Currie’s formulation led to an American Chemical Society symposium on the topic [3] and has been incorporated in many rules of practice governing measurement procedures, international standards [4], regulations, and software. The culmination of Currie’s early work was seen in the adoption of a harmonized international position (ISO-IUPAC) on the nomenclature, concepts, and formulation of detection, decision, and detection limits [5, 6].

In order to foster consistency of data reporting in all of science, in 1993 the International Organization for Standardization (ISO) issued its Guide to the Expression of Uncertainty in Measurement (GUM) [7] and the International Vocabulary of Basic and General Terms in Metrology (VIM) [8]. Directed toward professional metrologists, these publications incorporated some concepts unfamiliar to practicing laboratory workers. Several organizations, e.g., NIST [9], Eurachem/CITAC [10], IAEA [11], and BIPM [12], published supplemental interpretations with examples appropriate to their fields. The intent of these standards is to foster the publication of reliable data, quantitatively and unambiguously traceable to the International System of Units (SI).

Data quality in practice

In practice, of course, not all routine measurements require detailed uncertainty evaluation every time: the effort must be proportional to its importance, following the principle of fitness for use. When numbers are to be published and used by others, however, Eisenhart’s admonitions apply despite the effort required. “The concepts of traceability are not always well accepted by the analytical chemistry community. There is a benign kind of neglect towards these ideas or even straight hostility.’ [13] Part of the reason is that a full ISO-compliant treatment of uncertainties requires differentiation of the measurement equation, which may be difficult. For example, extracting the derivative of the equation used to reduce counting data to standard conditions

$$A_0=\frac{C\lambda e^{\lambda t}{e}^{P\delta /\mathit{\Delta }}(e^{\lambda \mathit{\Delta }}-1)}{(1-e^{-\lambda \mathit{\Delta }})(1-e^{-\lambda \tau })\lambda \mathit{\Delta }}$$

with respect to λ is daunting to most people. However, numerical re-calculations of the measurement equation employing finite differences instead of derivatives [14, 15, 10] can quantitatively show the effect upon the uncertainty of the final result due to uncertainties in the parameters. Applied to neutron activation analysis [16–18], this approach clearly shows the relevant uncertainties.

The two classic 1968 papers [1, 2] and the ISO GUM set the standards for clarity in future publications, yet we still see many papers submitted for review and even published with uncertainties given as simply “±” or omitted completely, or with many more digits than are truly significant. As Eisenhart pointed out, these practices have consequences for the users of these papers. As an example from our own work, a newly published half-life for ⁷⁶As [19] was accepted by the evaluators because of a plausible (but novel and incomplete) description of the uncertainties in the measurement, even though this value conflicted with previous measurements in the literature (notably a set of seven determinations with six different kinds of detectors [20]). This half-life would have led to an inaccurate INAA value for Arsenic Implant in Silicon (SRM 2134) had we used it in our measurements [21]. A redetermination of the half-life [22, 23] in agreement with the previous consensus is slowly driving out the incorrect value from tabulations of nuclear data.

Data quality and good science

Although science is always open to new ideas, misunderstanding the real uncertainties in a measurement, in physics as well as statistics, has led to some conspicuously mistaken conclusions. Scientific judgment (prior probability, in Bayesian terms) needs to be applied, especially to unexpected observations, and the null hypothesis that the observation is in error needs to be explicitly tested. Perhaps the best-known recent example is cold fusion [24] where, in the rush to publish, blank experiments were not adequately done and people knowledgeable in nuclear science were not consulted. Early measurements in our laboratory at NIST [25] without the Pd/D electrochemical cell showed neutrons (from cosmic rays) and gammas (mostly from ²¹⁴Bi) in quantitative agreement with Fleischmann and Pons’s paper. Other workers were also unable to duplicate the publicized work electrochemically, so that research in low-energy nuclear reactions (LENR) has nearly, but not entirely, stopped.

Since the discovery of radioactivity, more than eighty attempts have been made to influence the rate of radioactive decay. None has had an effect, with the sole exception of decay modes that involve the orbital electrons, the physics of which is well understood [26]. More recently, an unexplained anomaly in the decay curve of ⁵⁴Mn was observed to be coincident with a strong solar flare, and statistical anomalies in other decay measurements made at Brookhaven National Laboratory and at the Physikalisch-Technische Bundesanstalt (PTB) were found to have annual periodicity [27, 28]. The cause was hypothesized to be related to solar neutrinos affecting the value of the decay constant.

To test this connection, measurements were done in our laboratory at NIST to compare the decay rates of paired intense sources of ¹⁹⁸Au with greatly different surface/volume ratios (sphere vs. foil or wire), and thus greatly different internal antineutrino fluxes [29, 30]. The half-lives of the paired sources were found to be indistinguishable, contrary to the prediction of the neutrino hypothesis. Decay rate measurements at Delft in the presence of neutrinos from the HOR reactor were found to be no different when the reactor was shut down [31], casting further doubt on the hypothesis. Other arguments degrade the solar connection [32]; for example, a ¹³⁷Cs source decayed at the expected rate as the MESSENGER spacecraft traveled from earth to Mercury’s orbit at 0.4 AU [33]. Recent measurements at PTB have caused most of the earlier anomalies to disappear as more sources of experimental bias have been revealed and eliminated [34, 35]. Half-life measurement is subject to many sources of bias, not all of which are readily detected [36]. Although there is decreasing evidence for non-constant radioactive decay rates, the search continues in some laboratories.

The surprising observation that light is emitted from collapsing bubbles produced by ultrasonic agitation of water has led to the hypothesis that the energy of the collapse might be sufficient to cause nuclear fusion, a process called sonofusion. Indeed, both tritium and neutrons were claimed to be detected in deuterated acetone under cavitation [37]. Other laboratories failed to duplicate these measurements, and the pursuit quickly collapsed as a combination of self-deception and fraud was revealed. Inspired by this, another group searched for nuclear transformations in sonicated water, and found increased amounts of uranium and even transuranic nuclides [38].

Subsequently [39] it was claimed that neutrons generated in the ultrasonic probe caused measurable changes in element concentrations. That work was accepted for publication with the editor’s comment “This paper has been evaluated by six peers and has been considered to contain questionable results. However, as it reports on results difficult to prove but indisputably important if correct, the editor takes full responsibility for making it public.” Other observations led the same group to believe that cavitation increases the rate of alpha decay of ²²⁸Th [40]. The physical evidence has been sharply criticized [41], and more reasonable explanations of the observation have been proposed [42].

More recently, piezonuclear transmutations of ⁶³Cu to ⁶⁵Zn via multiple neutron capture were claimed [43], even though gamma spectrometry showed that the ultrasonic probe itself was no more radioactive after operation than before. The publication of this report resulted in strong criticism [44, 45] from the neutron activation analysis community, with the admonition, echoing others [41, 42], that the reviewers and editors of journals have an important responsibility to see that only verifiable facts and theories appear in the published literature.

Error and bias have many ways to creep into laboratory measurements. Currie (pers. comm., 1990) has pointed out that in a real measurement process “d.f. < 0 always; since the number of variables exceeds the number of observations, scientific insight is essential.” Anomalous observations may indeed point to new phenomena, but simple explanations are usually most probable. “The first principle is that you must not fool yourself — and you are the easiest person to fool.” (R. P. Feynman, 1974 Caltech commencement address).

Summary and conclusions

Science is a communal activity whose practitioners build upon each others’ work. To exploit the literature we must understand its limitations, which is possible only if the authors of publications understand the uncertainties in their measurements and conclusions, and make us, the readers, understand them in the same way.