Skip to main content
PMC home page
International Journal of Methods in Psychiatric Research logoLink to International Journal of Methods in Psychiatric Research
. 2008 Jan 10;16(4):202–207. doi: 10.1002/mpr.225

Inconsistencies between reported test statistics and p‐values in two psychiatry journals

David Berle 1,✉, Vladan Starcevic 2
PMCID: PMC6878403  PMID: 18188836

Abstract

A recent survey of the British Medical Journal (BMJ) and Nature revealed that inconsistencies in reported statistics were common. We sought to replicate that survey in the psychiatry literature. We checked the consistency of reported t‐test, F‐test and χ 2‐test values with their corresponding p‐values in the 2005 issues of the Australian and New Zealand Journal of Psychiatry (ANZJP) and compared this with the issues of the ANZJP from 2000, and with a similar journal, Acta Psychiatrica Scandinavica (APS). A reported p‐value was ‘inconsistent’ if it differed (at its reported number of decimal places) from our calculated p‐values (using three different software packages), which we based on the reported test statistic and degrees of freedom. Of the 546 results that we checked, 78 (14.3%) of the p‐values were inconsistent with the corresponding degrees of freedom and test statistic. Similar rates of inconsistency were found in APS and ANZJP, and when comparing the ANZJP between 2000 and 2005. The percentages of articles with at least one inconsistency were 8.5% for ANZJP 2005, 9.9% for ANZJP 2000 and 12.1% for APS. We conclude that inconsistencies in p‐values are common and may reflect errors of analysis and rounding, typographic errors or typesetting errors. Suggestions for reducing the occurrence of such inconsistencies are provided. Copyright © 2008 John Wiley & Sons, Ltd.

Keywords: statistics, results, conclusions, errors, inconsistencies

Full Text

The Full Text of this article is available as a PDF (132.3 KB).

REFERENCES

  1. Altman DG, Bland JM. Improving doctors' understanding of statistics. J R Stat Soc (Ser A) 1990; 154: 223–67. [Google Scholar]
  2. Altman DG, Schulz KF, Moher D, Egger M, Davidoff F, Elbourne D, Gotzsche PC, Lang T. The revised CONSORT statement for reporting randomized trials: explanation and elaboration. Ann Int Med 2001; 134: 663–94. [DOI] [PubMed] [Google Scholar]
  3. American Psychological Association . Publication Manual. Washington, DC: American Psychological Association, 2001. [Google Scholar]
  4. Avram MJ, Shanks CA, Dykes MH, Ronai AK, Stiers WM. Statistical methods in anesthesia articles: an evaluation of two American journals during two six‐month periods. Anesth Analg 1985; 64: 607–11. [PubMed] [Google Scholar]
  5. Balon R. By whom and how is the quality of research data collection assured and checked? Psychother Psychosom 2005; 74: 331–35. [DOI] [PubMed] [Google Scholar]
  6. Cohen J. The earth is round (p < 0.05). Am Psychol 1994; 49: 997–1003. [Google Scholar]
  7. Cruess DF. Review of the use of statistics in the American Journal of Tropical Medicine and Hygiene for January–December 1988. Am J Trop Med Hyg 1989; 41: 619–26. [DOI] [PubMed] [Google Scholar]
  8. Cumming G, Finch S. A primer on the understanding, use, and calculation of confidence intervals that are based on central and noncentral distributions. Educ Psychol Meas 2001; 61: 532–74. [Google Scholar]
  9. Dallal GE. Statistical microcomputing – like it is. Am Stat 1988; 42: 212–16. [Google Scholar]
  10. Davies J. A critical survey of scientific methods in two psychiatry journals. Aust N Z J Psychiatry 1987; 21: 367–73. [DOI] [PubMed] [Google Scholar]
  11. Emerson JD, Colditz GA. Use of statistical analysis in the New England Journal of Medicine . N Engl J Med 1983; 309: 709–13. [DOI] [PubMed] [Google Scholar]
  12. Felson DT, Cupples LA, Meenan RF. Misuse of statistical methods in Arthritis and Rheumatism. 1982 versus 1967–68. Arthritis Rheum 1984; 27: 1018–22. [DOI] [PubMed] [Google Scholar]
  13. García‐Berthou EC, Alcaraz C. Incongruence between test statistics and P values in medical papers. BMC Med Res Methodol 2004; 4: 13. [DOI] [PMC free article] [PubMed] [Google Scholar]
  14. Gore SM, Jones IG, Rytter EC. Misuse of statistical methods: critical assessment of articles in BMJ from January to March 1976. BMJ 1977; 1: 85–7. [DOI] [PMC free article] [PubMed] [Google Scholar]
  15. Hall W, Bird KD. The problem of multiple inference in psychiatric research. Aust N Z J Psychiatry 1985; 19: 265–74. [DOI] [PubMed] [Google Scholar]
  16. Hand D, Sham P. Improving the quality of statistics in psychiatric research. Br J Psychiatry 1995; 167: 689–91. [DOI] [PubMed] [Google Scholar]
  17. Hand DJ. The role of statistics in psychiatry. Psychol Med 1985; 15: 471–6. [DOI] [PubMed] [Google Scholar]
  18. Knüsel L. On the accuracy of statistical distributions in Microsoft Excel 97. Comput Stat and Data Anal 1988; 26: 375–7. [Google Scholar]
  19. MacArthur RD, Jackson GG. An evaluation of the use of statistical methodology in the Journal of Infectious Diseases . J Infect Dis 1984; 149: 349–54. [DOI] [PubMed] [Google Scholar]
  20. McCullough BD. Assessing the reliability of statistical software: Part I. Am Stat 1998; 52: 358–66. [Google Scholar]
  21. McGuigan SM. The use of statistics in the British Journal of Psychiatry . Br J Psychiatry 1995; 167: 683–8. [DOI] [PubMed] [Google Scholar]
  22. Murphy JR. Statistical errors in immunologic research. J Allergy Clin Immunol 2004; 114: 1259–63. [DOI] [PubMed] [Google Scholar]
  23. Olsen CH. Review of the use of statistics in Infection and Immunity . Infect Immun 2003; 71: 6689–92. [DOI] [PMC free article] [PubMed] [Google Scholar]
  24. Perneger TV. What's wrong with Bonferroni adjustments. BMJ 1998; 316: 1236–8. [DOI] [PMC free article] [PubMed] [Google Scholar]
  25. Vrbos LA, Lorenz MA, Peabody EH, McGregor M. Clinical methodologies and incidence of appropriate statistical testing in orthopaedic spine literature. Are statistics misleading? Spine 1993; 18: 1021–9. [DOI] [PubMed] [Google Scholar]
  26. Wright DB. Making friends with your data: improving how statistics are conducted and reported. Br J Educ Psychol 2003; 73: 123–36. [DOI] [PubMed] [Google Scholar]

Articles from International Journal of Methods in Psychiatric Research are provided here courtesy of Wiley

RESOURCES