Skip to main content

Some NLM-NCBI services and products are experiencing heavy traffic, which may affect performance and availability. We apologize for the inconvenience and appreciate your patience. For assistance, please contact our Help Desk at info@ncbi.nlm.nih.gov.

Journal of Applied Behavior Analysis logoLink to Journal of Applied Behavior Analysis
. 1977 Spring;10(1):133–139. doi: 10.1901/jaba.1977.10-133

Observer agreement, credibility, and judgment: some considerations in presenting observer agreement data

Thomas R Kratochwill 1,1, Ralph J Wetzel 1
PMCID: PMC1311160  PMID: 16795542

Abstract

Graphical and statistical indices employed to represent observer agreement in interval recording are described as “judgmental aids”, stimuli to which the researcher and scientific community must respond when viewing observer agreement data. The advantages and limitations of plotting calibrating observer agreement data and reporting conventional statistical aids are discussed in the context of their utility for researchers and research consumers of applied behavior analysis. It is argued that plotting calibrating observer data is a useful supplement to statistical aids for researchers but is of only limited utility for research consumers. Alternatives to conventional per cent agreement statistics for research consumers include reporting special agreement estimates (e.g., per cent occurrence agreement and nonoccurrence agreement) and correlational statistics (e.g., Kappa and Phi).

Keywords: observational data, methodology, observer bias, observer training, reliability, validity, experimenter calculations

Full text

PDF
133

Selected References

These references are in PubMed. This may not be the complete list of references from this article.

  1. Hartmann D. P. Considerations in the choice of interobserver reliability estimates. J Appl Behav Anal. 1977 Spring;10(1):103–116. doi: 10.1901/jaba.1977.10-103. [DOI] [PMC free article] [PubMed] [Google Scholar]
  2. Johnson S. M., Lobitz G. K. Parental manipulation of child behavior in home observations. J Appl Behav Anal. 1974 Spring;7(1):23–31. doi: 10.1901/jaba.1974.7-23. [DOI] [PMC free article] [PubMed] [Google Scholar]
  3. Kent R. N., O'Leary K. D., Diament C., Dietz A. Expectation biases in observational evaluation of therapeutic change. J Consult Clin Psychol. 1974 Dec;42(6):774–780. doi: 10.1037/h0037516. [DOI] [PubMed] [Google Scholar]
  4. Michael J. Statistical inference for individual organism research: mixed blessing or curse? J Appl Behav Anal. 1974 Winter;7(4):647–653. doi: 10.1901/jaba.1974.7-647. [DOI] [PMC free article] [PubMed] [Google Scholar]
  5. O'leary K. D., Kent R. N., Kanowitz J. Shaping data collection congruent with experimental hypotheses. J Appl Behav Anal. 1975 Spring;8(1):43–51. doi: 10.1901/jaba.1975.8-43. [DOI] [PMC free article] [PubMed] [Google Scholar]

Articles from Journal of Applied Behavior Analysis are provided here courtesy of Society for the Experimental Analysis of Behavior

RESOURCES