Consensus-based Construction of a Taxonomy of Clinical Trial Tasks

Abstract

The use of conceptual knowledge collections, such as taxonomies, is prevalent throughout the biomedical domain. Tools and methods to enable the computational representation of such knowledge collections are well established. However, methods for the design and construction of knowledge collections are varied and often rely on the judgments of a single, or small group of, specially trained knowledge engineers. This poster will report on a novel multi-expert consensus-based approach to the development of taxonomies that has been applied to the development of a taxonomy of clinical trial tasks and events.

Introduction

The use of conceptual knowledge collections within the biomedical domain is pervasive, spanning a broad spectrum from controlled terminologies to clinical decision support systems. A common approach to the modeling of such knowledge collections is the construction of taxonomies – explicit structures containing both atomic units of knowledge and hierarchical relationships between those units. Numerous tools and methods exist to facilitate the representation of taxonomies in computable formats. In contrast, methods for the initial construction of taxonomies are less well established. The few formal methods that have been reported upon rely heavily on the judgments of individual knowledge engineers with specific domain knowledge [1] – a scenario that may not be realistic in many specialized domains. This poster describes a pilot study of a novel method used in the construction of a taxonomy of clinical trial tasks and events. The method was designed to incorporate both semi-automated and multi-expert-based knowledge acquisition techniques, maximizing domain coverage and design efficiency. This method is motivated by the need for more generally accessible and interpretable taxonomies within the biomedical domain [2].

Methodology

The method used during the design of the clinical trial task taxonomy consists of three phases. In the first phase, clinical trial tasks and events are extracted from protocol documents and mapped to a standard nomenclature. In the second phase, common tasks or events are selected based upon a composite concept, protocol, and treatment-group level support metric. In the third phase, domain experts sort the selected tasks and events into groups, which are then analyzed using hypothesis discovery tools in order to develop a consensus-based taxonomy.

Results

During a pilot study of the described methodology, 32 protocol schemas from five treatment groups were abstracted, producing 522 task or event instances that when mapped to the UMLS constituted 93 unique concepts. The composite support metric described earlier was calculated, and the 65 concepts that fell within the 95^th percentile of a rank-ordered distribution of the metric’s distribution were selected for subsequent analysis. Three domain experts performed an all-in-one sort of the set of 65 concepts. Aggregate agreement of the experts concerning group composition was 58 ± 16%. A computational simulation of comparable, random sorting behavior predicted an aggregate agreement of 8 ± 1%. Hierarchical clustering analysis of the sorting results generated 16 unique clusters of concepts that were found to provide a reasonable basis for constructing a taxonomy.

Conclusions

Our initial results indicate that the proposed method for the consensus-based construction of a taxonomy, in this case encompassing clinical trials tasks and events, is both tractable and demonstrates quantitatively measurable characteristics that enable the verification and validation of the resulting knowledge collection.