The Certainty-Agreement Diagram: Comparing the Functionality of Coding Schemes in Primary Care Clinical Information Systems

Introduction

Despite enormous investment of effort and resources there are few formal comparative evaluations between different coding schemes. We have recently described a methodology of a randomised crossover trial comparing the performance of Clinical Terms Version 3 (CTV3) and Read Codes 5 Byte set (RC5B) coding schemes in General Practice. ¹ This study looked at the comparative performance of the two coding schemes by 10 general practitioners using a total of 995 concepts extracted from clinical records. This paper describes the use of the Certainty-Agreement diagram in providing a graphical representation of comparative functionality.

Certainty-Agreement Diagram

Biological and social systems are inherently complex, and by implication so are the coding schemes that have to capture this data. The certainty-agreement diagram (figure 1) can be used to estimate whether the issue is simple (high certainty, high agreement), chaotic (low certainty, low agreement), or complex (intermediate levels of one or both). ²

Figure 1.

Certainty-Agreement Diagram ^{3, 4}

Method

Ten general practitioners (GPs) in five matched pairs coded a series of concepts extracted from clinical records using both RC5B and CTV3. Two clinicians independently ranked whether the coded choice were exact (or non-exact). The certainty (average proportion of exact matches) and agreement (proportion with the same code chosen) between the 5 paired GPs were identified.

Results

The following diagram (figure 2) illustrates that there is significant improvement in agreement and certainty of a GPs ability to code using CTV3 compared to RC5B.

Figure 2.

Certainty (Proportion exact) -Agreement Diagram of coding between matched doctors using CTV3 & RC5B

Summary

The performance of different coding schemes can be helpfully illustrated with the use of a certainty-agreement diagram.