Abstract
Previous recommendations to employ occurrence, nonoccurrence, and overall estimates of interobserver reliability for interval data are reviewed. A rationale for comparing obtained reliability to reliability that would result from a random-chance model is explained. Formulae and graphic functions are presented to allow for the determination of chance agreement for each of the three indices, given any obtained per cent of intervals in which a response is recorded to occur. All indices are interpretable throughout the range of possible obtained values for the per cent of intervals in which a response is recorded. The level of chance agreement simply changes with changing values. Statistical procedures that could be used to determine whether obtained reliability is significantly superior to chance reliability are reviewed. These procedures are rejected because they yield significance levels that are partly a function of sample sizes and because there are no general rules to govern acceptable significance levels depending on the sizes of samples employed.
Keywords: interobserver reliability, interval data, statistical inference, chance agreement, reliability criteria
Full text
PDF
Selected References
These references are in PubMed. This may not be the complete list of references from this article.
- Ayllon T., Roberts M. D. Eliminating discipline problems by strengthening academic performance. J Appl Behav Anal. 1974 Spring;7(1):71–76. doi: 10.1901/jaba.1974.7-71. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Bijou S. W., Peterson R. F., Ault M. H. A method to integrate descriptive and experimental field studies at the level of data and empirical concepts. J Appl Behav Anal. 1968 Summer;1(2):175–191. doi: 10.1901/jaba.1968.1-175. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Glynn E. L., Thomas J. D. Effect of cueing on self-control of classroom behavior. J Appl Behav Anal. 1974 Summer;7(2):299–306. doi: 10.1901/jaba.1974.7-299. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Knapczyk D. R., Livingston G. The effects of prompting question-asking upon on-task behavior and reading comprehension. J Appl Behav Anal. 1974 Spring;7(1):115–121. doi: 10.1901/jaba.1974.7-115. [DOI] [PMC free article] [PubMed] [Google Scholar]