Observer accuracy in estimating proportions in images: implications for the semiquantitative assessment of staining reactions and a proposal for a new system

Abstract

Background—The results of immunohistochemical staining are often assessed by semiquantitative scoring. However, these scoring systems are usually non-standardised and there has been little evaluation of the accuracy and reliability of this subjective assessment.

Aims—To assess the accuracy of observer estimation of proportions of objects in an image.

Methods—Images were generated that contained known proportions of pink squares in grids of 50 x 50 and 100 x 100 squares. Observers were shown each image for five seconds in random order and either estimated the proportion of pink squares or selected the image (from a pair of images) that contained the greater proportion of pink squares. The observers were four consultant histopathologists, seven trainee histopathologists, and six control non-histopathologists.

Results—The raw estimations of proportions showed a close correlation with the real proportions, with correlation coefficients of 0.94 and 0.95 for consultant and trainee histopathologists on the 50 x 50 grids. However, the performance in the comparison task was much higher, with an almost perfect classification for grids of equal size even when the proportions only differed by 5%.

Conclusions—Histopathologists can estimate proportions of objects in an image with a reasonable degree of accuracy in this abstract test system. All observers, whether histopathologists or not, can discriminate between proportions that are only 5% different in equal sized image grids. This suggests that the generation and use of carefully calibrated reference images could greatly improve the accuracy and reliability of semiquantitative scoring of immunohistochemical or any other staining.

Key Words: scoring • immunohistochemistry • semiquantitative • interobserver agreement • κ statistics

Full Text

The Full Text of this article is available as a PDF (288.2 KB).

Figure 1 Examples of the test images. (A) A 50 x 50 grid in which 40% of the squares are coloured pink. (B) Two 100 x 100 grids in a comparison image; the left hand grid has 50% pink squares, the right hand grid has 45%. Despite the small difference in the proportion of pink squares between the two grids all the observers correctly selected the left hand grid as the one with the greatest proportion. (C) Two grids in a comparison image. The left hand 100 x 100 grid has 50% pink squares, the right hand 50 x 50 grid has 55% pink squares. Most of the errors made in the entire study related to this image, although 74% of observers still correctly identified the right hand grid as containing the greatest proportion.

Figure 2 Scattergrams for the estimated proportion of pink squares in a 50 x 50 grid for (A) the consultant histopathologists, (B) the trainee histopathologists, and (C) controls. The regression line and correlation coefficient were derived using the least squares method of regression.

Figure 3 Scattergrams for the estimated proportion of pink squares in a 100 x 100 grid for (A) the consultant histopathologists, (B) the trainee histopathologists, and (C) controls. The regression line and correlation coefficient were derived using the least squares method of regression. It can be seen that the points have a greater scatter around the regression line than for the 50 x 50 grids (fig 2). This scatter is greater in the control group than in the consultant or trainee histopathologist groups and there is a tendency towards overestimation of the proportion.