Abstract
Graphical and statistical indices employed to represent observer agreement in interval recording are described as “judgmental aids”, stimuli to which the researcher and scientific community must respond when viewing observer agreement data. The advantages and limitations of plotting calibrating observer agreement data and reporting conventional statistical aids are discussed in the context of their utility for researchers and research consumers of applied behavior analysis. It is argued that plotting calibrating observer data is a useful supplement to statistical aids for researchers but is of only limited utility for research consumers. Alternatives to conventional per cent agreement statistics for research consumers include reporting special agreement estimates (e.g., per cent occurrence agreement and nonoccurrence agreement) and correlational statistics (e.g., Kappa and Phi).
Keywords: observational data, methodology, observer bias, observer training, reliability, validity, experimenter calculations
Full text
PDF![133](https://cdn.ncbi.nlm.nih.gov/pmc/blobs/f79c/1311160/3d5ab8ef9fae/jaba00112-0135.png)
![134](https://cdn.ncbi.nlm.nih.gov/pmc/blobs/f79c/1311160/c39e205034a2/jaba00112-0136.png)
![135](https://cdn.ncbi.nlm.nih.gov/pmc/blobs/f79c/1311160/d28cd4f6948e/jaba00112-0137.png)
![136](https://cdn.ncbi.nlm.nih.gov/pmc/blobs/f79c/1311160/b4d1d7ba26ad/jaba00112-0138.png)
![137](https://cdn.ncbi.nlm.nih.gov/pmc/blobs/f79c/1311160/af5717fe6a71/jaba00112-0139.png)
![138](https://cdn.ncbi.nlm.nih.gov/pmc/blobs/f79c/1311160/ef152e2120a6/jaba00112-0140.png)
![139](https://cdn.ncbi.nlm.nih.gov/pmc/blobs/f79c/1311160/0a6577822062/jaba00112-0141.png)
Selected References
These references are in PubMed. This may not be the complete list of references from this article.
- Hartmann D. P. Considerations in the choice of interobserver reliability estimates. J Appl Behav Anal. 1977 Spring;10(1):103–116. doi: 10.1901/jaba.1977.10-103. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Johnson S. M., Lobitz G. K. Parental manipulation of child behavior in home observations. J Appl Behav Anal. 1974 Spring;7(1):23–31. doi: 10.1901/jaba.1974.7-23. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kent R. N., O'Leary K. D., Diament C., Dietz A. Expectation biases in observational evaluation of therapeutic change. J Consult Clin Psychol. 1974 Dec;42(6):774–780. doi: 10.1037/h0037516. [DOI] [PubMed] [Google Scholar]
- Michael J. Statistical inference for individual organism research: mixed blessing or curse? J Appl Behav Anal. 1974 Winter;7(4):647–653. doi: 10.1901/jaba.1974.7-647. [DOI] [PMC free article] [PubMed] [Google Scholar]
- O'leary K. D., Kent R. N., Kanowitz J. Shaping data collection congruent with experimental hypotheses. J Appl Behav Anal. 1975 Spring;8(1):43–51. doi: 10.1901/jaba.1975.8-43. [DOI] [PMC free article] [PubMed] [Google Scholar]