Skip to main content
NCBI home page
Journal of Applied Behavior Analysis logoLink to Journal of Applied Behavior Analysis
. 1982 Spring;15(1):189–190. doi: 10.1901/jaba.1982.15-189

A cautionary note on the use of probability values to evaluate interobserver agreement

Donald P Hartmann 1, William Gardner 1
PMCID: PMC1308259  PMID: 16795656

Abstract

Proposed methods of assessing the statistical significance of interobserver agreements provide erroneous probability values when conducted on serially correlated data. Investigators who wish to evaluate interobserver agreements by means of statistical significance can do so by limiting the analysis to every kth interval of data, or by using Markovian techniques which accommodate serial correlations.

Keywords: interobserver agreement, observational reliability, statistics, serial dependence

Full text

PDF
189

Selected References

These references are in PubMed. This may not be the complete list of references from this article.

  1. Birkimer J. C., Brown J. H. Back to basics: Percentage agreement measures are adequate, but there are easier ways. J Appl Behav Anal. 1979 Winter;12(4):535–543. doi: 10.1901/jaba.1979.12-535. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Hartmann C. Studies on the development of extra-endothelial and intra-endothelial pigment deposits by means of direct and indirect contact specular microscopy of the cornea. Graefes Arch Clin Exp Ophthalmol. 1982;218(2):75–82. doi: 10.1007/BF02153715. [DOI] [PubMed] [Google Scholar]
  3. Hartmann D. P. Considerations in the choice of interobserver reliability estimates. J Appl Behav Anal. 1977 Spring;10(1):103–116. doi: 10.1901/jaba.1977.10-103. [DOI] [PMC free article] [PubMed] [Google Scholar]
  4. Hartmann D. P., Gottman J. M., Jones R. R., Gardner W., Kazdin A. E., Vaught R. S. Interrupted time-series analysis and its application to behavioral data. J Appl Behav Anal. 1980 Winter;13(4):543–559. doi: 10.1901/jaba.1980.13-543. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. Hawkins R. P., Fabry B. D. Applied behavior analysis and interobserver reliability: A commentary on two articles by Birkimer and Brown. J Appl Behav Anal. 1979 Winter;12(4):545–552. doi: 10.1901/jaba.1979.12-545. [DOI] [PMC free article] [PubMed] [Google Scholar]
  6. Hopkins B. L. Proposed conventions for evaluating observer reliability: A commentary on two articles by Birkimer and Brown. J Appl Behav Anal. 1979 Winter;12(4):561–564. doi: 10.1901/jaba.1979.12-561. [DOI] [PMC free article] [PubMed] [Google Scholar]
  7. Yelton A. R. Reliability in the context of the experiment: A commentary on two articles by Birkimer and Brown. J Appl Behav Anal. 1979 Winter;12(4):565–569. doi: 10.1901/jaba.1979.12-565. [DOI] [PMC free article] [PubMed] [Google Scholar]
  8. Yelton A. R., Wildman B. G., Erickson M. T. A probability-based formula for calculating interobserver agreement. J Appl Behav Anal. 1977 Spring;10(1):127–131. doi: 10.1901/jaba.1977.10-127. [DOI] [PMC free article] [PubMed] [Google Scholar]

Articles from Journal of Applied Behavior Analysis are provided here courtesy of Society for the Experimental Analysis of Behavior

RESOURCES