Bioanalytical inaccuracy: a threat to the integrity and efficiency of research

In recent years, the threats to the integrity of science posed by the dishonesty of a small minority of scientists,¹ and the biases held by most scientists,^2,3 have become a topic of lively discussion. We believe that faulty measurement of biochemical compounds is sufficiently common that it also serves to undermine the integrity and efficiency of science. We approach this neglected area by first discussing a series of erroneous reports in the neuroscience literature that we are personally familiar with owing to our areas of research interest. Then we attempt to explain the underlying factors that contribute to the problem and make suggestions about how to minimize and mitigate the problem.

Serotonin in human lumbar cerebrospinal fluid

A large number of papers have reported mean levels of serotonin in human lumbar cerebrospinal fluid (CSF) ranging from 0.02 ng/mL to more than 10 ng/mL. Many of the reported methods employed high-performance liquid chromatography (HPLC) with electrochemical (amperometric or coulometric) detection. Widely used because of its general applicability, HPLC involves separating compounds from one another based on their differing retention times on a column. Typically, a peak detected in a sample with the same retention time as a standard is assumed to consist solely of the standard compound. Unfortunately, this is not always the case. In the case of lumbar CSF serotonin, a study that used 2 selective and sensitive HPLC methods demonstrated that the actual concentration of serotonin in human lumbar CSF was less than 0.01 ng/mL.⁴ It is clear that reports of serotonin in human lumbar CSF at levels much above 0.01 ng/mL were in error, because of either substantial blood contamination or nonspecific analytical methods or both. Unfortunately, a number of such reports continued to appear in a variety of journals^5–13 well after the publication of the paper that raised fundamental concerns regarding human lumbar CSF serotonin measurements.⁴ It should be noted that serotonin can apparently be measured accurately in cisternal CSF of rodents¹⁴ and nonhuman primates.¹⁵

Serotonin in platelet-poor plasma

A similar situation exists with respect to serotonin levels in platelet-poor plasma (PPP), with the continuing publication of reports of erroneously high PPP serotonin concentrations despite 2 reports demonstrating that the true levels were less than 0.5 ng/mL and outlining the steps needed to obtain more accurate measurements.^16,17 Levels of PPP serotonin ranging from about 1 ng/mL to over 100 ng/mL have been reported by researchers from a wide range of fields, including biological psychiatry, neuropharmacology, cardiovascular medicine, endocrinology and gastroenterology. Few, if any, of the previous reports appear to be valid. Research in this area has become of greater interest owing to the recent recognition of the potential importance of free plasma serotonin in bone metabolism,¹⁸ placental and embryonic development¹⁹ and pancreatic function.²⁰ Assessment of the role of free plasma serotonin in these various realms will obviously require increased attention to analytical methods.

Tryptamine derivatives in the brain

Interest in another tryptophan metabolite, tryptamine, has fluctuated over the years but is likely to increase with the discovery of a family of mammalian G protein–coupled trace amine receptors.²¹ An important issue is the amount of tryptamine in the brain. A radioenzymatic method found 50 ng/g in rat striatum,²² but a later study involving gas chromatography–mass spectrometry (GC–MS) reported only 0.35 ng/g.²³ The most plausible explanation for the 100-fold discrepancy is that the radioenzymatic method did not selectively measure tryptamine. However, whereas mass spectrometric methods are often regarded as the most reliable and selective assays, GC–MS and liquid chromatographic–MS (LC–MS) methods are not infallible. For example, the “absolute determination” of the amount of 5-methoxytryptamine in rat hypothalamus was reported using a GC–MS method.²⁴ This was of interest because it suggested that serotonin, like catecholamines, could be O-methylated to an active metabolite. However, the measurement of 5-methoxytryptamine appears to have been an artifact due to the presence of melatonin (N-acetyl-5-methoxytryptamine) in rat hypothalamus. During sample preparation, melatonin was deacetylated to 5-methoxytryptamine.²⁵

Quinolinic acid in the brain

Another tryptophan metabolite not actually present in the brain at the levels sometimes reported is quinolinic acid. Accurate quantitation of endogenous levels of brain quinolinic acid is important because the compound can be neurotoxic when injected at high doses. Quinolinic acid was reported to be elevated in a rat model of hepatic encephalopathy.²⁶ In control animals, the brain content of quinolinic acid was reported to be about 1000 pmol per gram of tissue. A later study with a mean control value of 20 pmol per gram of tissue reported a decline in quinolinic acid levels in the same model.²⁷ Although both studies used MS methods, the 50-fold difference is most probably related to faulty analytical methods, emphasizing the need for rigorous validation of the method used in any initial report of a compound in a particular matrix.

Another study looked at the possibility that quinolinic acid could contribute to lesions in Huntington disease.²⁸ Quinolinic acid was reported to be elevated 3- to 4-fold in the postmortem brains of patients with early but not late stage Huntington disease. Quinolinic acid levels in control postmortem brains measured using a GC–MS method were reported to be about 2 μmol per milligram protein. Given that brain is about 10% protein, this would be 0.2 μmol per milligram of tissue or 0.2 mmol per gram of tissue. Three other studies all using GC–MS methods have reported quinolinic acid in human brain at levels between about 0.1 and 1 nmol per gram of tissue,^29–31 values that are smaller by a factor of about one million. It is doubtful that quinolinic acid was actually being measured in the study of Huntington disease patients;²⁸ at the very least, there are large errors in the units used to express the measured concentrations. Certainly, statements about similar concentrations causing neurotoxicity and claims that “The results revitalize the quinolinate hypothesis of HD …” should be dismissed.

Oxytocin and arginine vasopressin in human urine

A million-fold error was present in a recent report in the Proceedings of the National Academy of Sciences that clearly arose from the use of inappropriate analytical methods. Fries and colleagues³² reported that urinary excretion of oxytocin and arginine vasopressin (AVP) was reduced in a group of Russian and Romanian orphans exposed to severe early neglect. The paper has attracted considerable attention and has been cited over 60 times in the 4 years since its publication. Unfortunately, the HPLC–UV absorbance method used to analyze the peptides in urine does not have sufficient sensitivity or selectivity to measure either oxytocin or AVP in human urine. Previous studies using bioassays and mainly radioimmunoassays have found oxytocin levels of 10–50 pg/mL (6 reports^33–38) and AVP levels of 10–100 pg/mL (25 reports^39–63) in human urine, without much effect of age on excretion. The mean levels of about 20 and 50 μg/mg creatinine reported by Fries and colleagues² for oxytocin and AVP, respectively, are equivalent to about 10 and 25 μg/mL, given the typical excretion of creatinine of 320 mg/day and the typical volume of urine in 24 hours (500–600 mL) for 4 year olds.⁶⁴ The report of Fries and colleagues³² cites only a single prior study of urinary oxytocin or AVP, and the million-fold discrepancy between that study’s results and the oxytocin results presented is not mentioned. It is clear that the urinary AVP and oxytocin data of Fries and colleagues should be disregarded and that no meaningful statements regarding the neurobiological effects of early neglect can be made based on the study.

General considerations and recommendations

One important issue is why the obvious discrepancy between the previous literature and the results reported by Fries and colleagues³² was not detected during peer review. Unfortunately, research indicates that the standard of peer review is, in general, poor. When 607 reviewers for the BMJ were sent 3 test manuscripts each containing 9 major errors, the reviewers detected less than one-third, an average of only 2.58 errors.⁶⁵ A second important issue is why the original paper is left in the literature and continues to be cited at a high rate when a paper reporting the inaccuracy of the data and the inappropriate analytical methods has been published.⁶⁶ Unfortunately, this phenomenon appears fairly common and has been studied. A paper titled “How citation distortions create unfounded authority” concluded that “Through distortions in its social use that includes bias, amplification, and invention, citation can be used to generate information cascades resulting in unfounded authority of claims.”⁶⁷

Although it is not possible to determine the exact extent of the problem of analytical inaccuracy, it is our sense that the types of cases presented in this article are not that rare, that less egregious cases are not uncommon, and that inaccuracies in general contribute in an important way to inefficiency in science. We also believe that these examples illustrate several factors that contribute to the problem of analytical inaccuracy. A major factor is the failure to validate methods carefully when they are first introduced or implemented. This failure can often be attributed to a lack of sufficient analytical training. Researchers require a large knowledge base covering a variety of disciplines. Among those, knowledge in statistics and analytical methods is often sadly lacking. Altman⁶⁸ has pointed out that research papers commonly contain methodological errors, report results selectively and draw unjustified conclusions. In particular, serious statistical errors are common even in some high-profile journals.⁶⁸ The problems with both statistics and analytical methods are probably related both to inadequate training in these topics and a lack of awareness on the part of some researchers about the limits of their own knowledge. Students learn from their supervisors to apply statistical and analytical methods without the theoretical or practical training that is essential to use them appropriately.

Another factor is the availability of kits and methods that seem to promise results without requiring much, if any, understanding of the physiochemical basis or limitations of the method. In addition, as illustrated in this article, ignorance or disregard of relevant prior research can contribute to researchers reporting erroneous data. A lack of healthy skepticism can lead others to persist in citing erroneous reports. Finally, difficulties in obtaining expert review of manuscripts from researchers with expertise in analytical methods may lead to errors passing through the review stage.

The causes of inaccurate assays are varied, ranging from genuine mistakes of the type that all researchers make from time to time, to gross incompetence and disregard of the need to strive for high standards. A number of steps should be taken to remedy this situation.

• All researchers using analytical methods should receive training either formally or informally in the principles of analytical methodology and the fundamentals of quality assessment and control.

• Research training should emphasize the need to do adequate literature reviews to determine whether a researcher’s results are consistent with published data as far as the quantitation of compounds is concerned. The published results establishing the validity of an assay should be included in the methods section of a manuscript for all but the most routine assays.

• If there is a discrepancy between the values obtained and the published levels of a compound, steps need to be taken to determine which result is accurate using the most rigorous analytical methods.

• Reviewers and journal editors need to do a better job of checking results in manuscripts against previously published results with respect to the analytical methods used and the absolute concentrations reported.

• Journals need to select reviewers based both on their knowledge of the subject matter of a paper under review and on their expertise with the analytical methods used.

Lack of sufficient attention to analytical methodology leads to the publication of inaccurate results, which in turn misleads researchers, decreases scientific efficiency and wastes taxpayers’ and foundations’ money.