Figure 1.

Phases of terminology development. Solid squares represent rating steps, involving sentences or concept-relation-concept tuples, one or more raters and (except in Phase 1) a version of the terminology. In Phase 1, sentences were taken from the New England Journal of Medicine (NEJM) and coded by one of the authors (JJC) and 7 fellows of the American College of Medical Informatics (ACMI) into 1155 tuples, from which JJC drew the initial terminology. In Phase 2, one of the authors (ZL) reviewed a 147-tuple subset of the 1155 tuples to identify terms that could be merged into the revised terminology. In Phase 3, a review of the entire set of 1155 tuples resulted in further term mergers and additions to produce the final terminology. In Phase 4, the final terminology was used by JJC, ZL and three clinical informaticians to code 51 sentences drawn from the Journal of Clinical Case Studies. Interrater reliability statistics for each phase are reported in Table 1.