Figure 1. Receiver operator characteristic (ROC) plot for all the graders who took part in the external arbitration.

The three levels of grader (primary, secondary and arbitration) are shown by ‘*’, ‘+’ and ‘o’ respectively. All the graders were compared against a consensus grading calculated from the results from all the arbitration level graders. Notice even amongst the arbitration level graders there is a range of operating points from high specificity and lower sensitivity to high sensitivity and lower specificity.