Inter-observer agreement in myelodysplastic syndromes

We read with great interest the recent paper by Senent et al.¹ and the accompanying editorial² on the reproducibility of WHO 2008 criteria and inter-observer agreement regarding the assessment of dysplasia for diagnosis and classification of myelodysplastic syndromes (MDS). The reproducibility of any composite criteria is mainly based on both the constituting components and the precise manner in which they are actually handled. The WHO 2008 classification is based on the quantification of bone marrow (BM) and peripheral blood blasts, degree of dyshemopoietic findings, precise peripheral blood cytopenias actually observed, presence or absence of Auer rods, as well as some cytogenetic features (i.e. the presence of 5q abnormalities), the way they must be handled being set out in the original WHO document.

The results from Senent and co-workers’ study of 50 MDS cases are based on a “blind and independent microscopical review by four cytologists from three centers”.¹ Their paper has a very high methodological quality and confirms our group’s 1999 findings³ based on 26 MDS cases and 24 control patients collected in one single center and performed by “five independent observers in a blinded manner”;³ only one of them was actually a cytologist, the others being either clinical hematologists or hematologists in training. Obviously, the number of observers, their cytological expertise, as well as the number of participating centers may have a role in the degree of inter-observer agreement but, interestingly enough, their results on individual cytological features do not seem to differ from our earlier work, and cast doubt on whether the observer’s expertise or, conversely, the subjectivity of the pathological findings concerned are the main contributors to the observed heterogeneity. As stressed in John Bennnett‘s editorial,² two main cytological features, namely BM blast cells and the proportion of pathological sideroblasts, are the most reproducible data in MDS patients, whereas the degree of dyserythropoietic findings, excluding sideroblasts, is the least reproducible. In addition, Senent and coworkers also showed that the concordance of the cut-off point for granulocytic (and megakaryocytic) lineage expressed by the kappa statistic is much higher for the 40% figure than for the WHO-based 10% value (kappa 0.40 vs. 0.19). Again, this agrees with our previous findings that showed that the percentage of dysgranulopoietic findings in control patients might be much higher than 10% (p90: 27%); something that suggests that the WHO 10% cut-off was not a good choice.

Finally, despite the quality of the data presented in Senent s paper, some questions remain to be addressed in the future: the inter-observer agreement of BM blasts was uncomfortably low when the observed proportion was located between 2% and 10%, a fact that may explain the discordance between their paper and other recently published work⁴ based on 100 MDS cases “collected from 10 hospitals and…evaluated by 10 morphologists, working in five pairs”: inter-observer concordance was lower in RAEB 1 in the former but maximal in the latter: the classification concordance (kappa statistic) was very low for refractory anemia with ringed sideroblasts (0.26, P=0.09) despite the great reproducibility of ring sideroblasts, something likely related to the contribution of the other components of the WHO classification, suggesting that a full and complete classification is yet to be achieved.