Rule-based support system for multiple UMLS semantic type assignments

Abstract

Background

When new concepts are inserted into the UMLS, they are assigned one or several semantic types from the UMLS Semantic Network by the UMLS editors. However, not every combination of semantic types is permissible. It was observed that many concepts with rare combinations of semantic types have erroneous semantic type assignments or prohibited combinations of semantic types. The correction of such errors is resource-intensive.

Objective

We design a computational system to inform UMLS editors as to whether a specific combination of two, three, four, or five semantic types is permissible or prohibited or questionable.

Methods

We identify a set of inclusion and exclusion instructions in the UMLS Semantic Network documentation and derive corresponding rule-categories as well as rule-categories from the UMLS concept content. We then design an algorithm adviseEditor based on these rule-categories. The algorithm specifies rules for an editor how to proceed when considering a tuple (pair, triple, quadruple, quintuple) of semantic types to be assigned to a concept.

Results

Eight rule-categories were identified. A Web-based system was developed to implement the adviseEditor algorithm, which returns for an input combination of semantic types whether it is permitted, prohibited or (in a few cases) requires more research. The numbers of semantic type pairs assigned to each rule-category are reported. Interesting examples for each rule-category are illustrated. Cases of semantic type assignments that contradict rules are listed, including recently introduced ones.

Conclusion

The adviseEditor system implements explicit and implicit knowledge available in the UMLS in a system that informs UMLS editors about the permissibility of a desired combination of semantic types. Using adviseEditor might help accelerate the work of the UMLS editors and prevent erroneous semantic type assignments.

1 Introduction

The Unified Medical Language System (UMLS) [1–4], is derived from about 160 source terminologies. Its Metathesaurus [5, 6] contains over two and a half million concepts. The UMLS Semantic Network (SN) [7–10] provides a compact semantic abstraction layer, consisting of 133 high-level, broad categories, called semantic types. One or more semantic types of the Semantic Network are assigned to each Metathesaurus concept, providing it with semantics, in the sense of describing the nature of the concept by identifying its one or more broad categories.

When there are two semantic types assigned to the same concept, a number of problems may occur. In some cases, one semantic type assignment may be redundant, because the other semantic type expresses the meaning of the concept in a more specific way. In other cases, one semantic type assignment may outright contradict another one, indicating an inconsistency in the UMLS semantic type assignments. These problems notwithstanding, multiple assignments are important to express fine shades of semantics. For some cases, e.g. for chemical concepts, multiple assignments are explicitly encouraged in the documentation of the UMLS Semantic Network. There is no public repository that expresses all the different legitimate ways of interplay between the 133 semantic types. Neither is there a complete list of prohibited combinations of semantic types.

When a concept is assigned multiple semantic types, it has compound semantics [11, 12], which is the combination of the semantics of the multiple semantic types. Such concepts are complex, due to their compound semantics of being simultaneously “this and that.” Our experience shows [11–15] that concepts with rare combinations of semantic types, i.e. there are only a few Metathesaurus concepts assigned exactly this combination, have a high likelihood of erroneous semantic type assignments. Furthermore, some semantic type assignments stand in contradiction to the explicit documentation of the UMLS Semantic Network. This situation suggests that UMLS editors would benefit from a support system, informing them regarding the permissibility of assigning a specific combination of semantic types to a concept.

The objective of this research is to develop a system adviseEditor that will inform an editor as to whether a specific tuple (pair, triple, quadruple, quintuple) of semantic types is permitted or prohibited. There is a need for such a system, because UMLS editors have introduced prohibited combinations of semantic types and even reintroduced them after the UMLS was corrected by eliminating those prohibited combinations. (Examples of such reintroduced combinations appear in Section 4.7.) To achieve this objective, we first need to define categories of rules that govern the possible interactions of pairs of semantic types. We will point out examples where concepts in the Metathesaurus violate the identified rules. If the adviseEditor system would have been in place when those concepts were originally introduced into the UMLS and assigned semantic types, these errors could have been prevented. We will also provide counts of semantic type pairs belonging to different rule-categories, as determined by the adviseEditor system.

2 Background

The Metathesaurus of the UMLS is the result of integrating about 160 source terminologies into one knowledge source. An important conceptual tool for this integration is the UMLS Semantic Network. Every concept in the Metathesaurus is assigned one or more semantic types of the Semantic Network at the time of integration [16, 17]. These assignments were performed by many UMLS editors at the National Library of Medicine over a long period of time, and thus are not necessarily done in a consistent manner.

The UMLS Semantic Network is structured as two separate trees, rooted in the semantic types Entity and Event, respectively. The 133 semantic types of the Semantic Network constitute its nodes and are connected by IS-A links. They are furthermore connected by 53 lateral relationship kinds. Inheritance of lateral relationships along IS-A links is by default a defined operation, except for a few cases where it is explicitly blocked.

When working with semantic types we make use of the following definition. Definition: The set of all concepts assigned a specific semantic type T is called the extent of T, abbreviated as E(T).

Whenever a concept is assigned two semantic types, then it is contained in the extents of both semantic types at the same time. Mathematically this means that the concept is in the set intersection of the two extents. The mathematical symbol ∩, expressing intersection, will occasionally be used when describing sets of concepts that are assigned two semantic types.

In [11, 12, 16] auditing of the UMLS for inconsistencies was carried out, based on intersections of extents of semantic types. We hypothesized [12] that concepts in small intersections have a high likelihood of wrong semantic type assignments. In a sample of 100 intersections, each containing only a single concept, analyzed by JJ Cimino [12], only 11 concepts were found to have correct semantic type assignments.

Gu et al. showed [17] that concepts assigned pairs of semantic types, such that the intersections of their extents are small, were more likely to have erroneous semantic type assignments than other concepts. In this paper, we make use of this observation for developing an algorithm for classifying pairs of semantic types according to rule-categories.

This research also builds on an algorithm [18] for identifying all redundant semantic type assignments, namely assignments in which a concept is assigned the semantic types X and Y such that X is a child or descendant of Y. Such redundant assignments are prohibited by the rules of the Semantic Network [19], and only X should be assigned. Assigning the respective pairs of semantic types is not legal, and they should never be assigned to the same concept. However, in the 1998 release we found 8622 concepts with redundant semantic type assignments in 77 prohibited intersections [12].

To help both editors and users of the UMLS, the National Library of Medicine provides a definition for each semantic type in the Semantic Network source data. Usage notes (UNs) are provided for some, but by far not all, semantic types. Note that in the balance of this paper, when we refer to a semantic type definition, we mean to include any usage notes attached to this definition. Some usage notes include instructions concerning the combination of two semantic types. These instructions describe situations in which a concept assigned one semantic type may not, may, or should be assigned a specific second semantic type.

3 Methods

3.1 Text-Based Instructions

Studying the documentation of the Semantic Network, one can distinguish between two kinds of instructions, inclusion instructions and exclusion instructions. An inclusion instruction expresses the fact that two semantic types may be used for the same concept or even should be used for the same concept. An exclusion instruction expresses the fact that two semantic types may not be used for the same concept.

We will use the semantic type Anatomical Abnormality to describe the following possible parts of a usage note: (1) specification, (2) inclusion instruction, and (3) exclusion instruction. Below is the UN provided in the UMLS about this semantic type.

UN: Use this type if the abnormality in question can be either an acquired or congenital abnormality. Neoplasms are not included here. These are given the type 'Neoplastic Process'. If an anatomical abnormality has a pathologic manifestation, then it will additionally be given the type 'Disease or Syndrome', e.g. “Diabetic Cataract” will be double-typed for this reason.

(1) Specification:

A specification may contain an additional explanation of what a certain semantic type stands for, or a set of requirements to be satisfied by a concept to be assigned this semantic type, or a clarification to distinguish between two semantic types.

In the above usage note of Anatomical Abnormality the following part corresponds to a specification. “Use this type if the abnormality in question can be either an acquired or congenital abnormality.”

In this case, one needs to realize that, as shown in Figure 1, Acquired Abnormality and Congenital Abnormality are the two children of Anatomical Abnormality in the Semantic Network. This specification instruction states that for an abnormality that can be of either kind, the more general parent semantic type Anatomical Abnormality should be assigned. This specification implies an exclusion instruction between the two children of Anatomical Abnormality.

Figure 1.

Anatomical Abnormality subhierarchy of SN

For example, the abnormalities “intestinal defect,” and “pharyngeal diverticulum” can be either acquired or congenital. Thus, the semantic type Anatomical Abnormality is assigned to them.

(2) Inclusion Instruction:

An inclusion instruction expresses the fact that two semantic types may be used for the same concept or even should be used for the same concept. In the above UN the following part corresponds to an inclusion instruction: “If an anatomical abnormality has a pathologic manifestation, then it will additionally be given the type 'Disease or Syndrome'.”

Thus, such a concept should be simultaneously assigned Anatomical Abnormality and Disease or Syndrome. Indeed, the Metathesaurus contains 940 concepts that are assigned these two semantic types, for example, Dynamic subaortic stenosis. In the Venn diagram in Figure 2, the intersection of extents of concepts, which are assigned Anatomical Abnormality and Disease or Syndrome, is marked by an “x.”

Figure 2.

The extent of Disease or Syndrome intersects the extent of Anatomical Abnormality and the extents of its two children

(3) Exclusion Instruction:

An exclusion instruction expresses the fact that two semantic types may not be used for the same concept. In the above usage note of Anatomical Abnormality the following part corresponds to an exclusion instruction: “Neoplasms are not included here. These are given the type Neoplastic Process.” Hence this exclusion instruction states that no concept is assigned both Anatomical Abnormality and Neoplastic Process. Thus, the concept conjunctival erosion is assigned Anatomical Abnormality. On the other hand, small cell carcinoma of prostate is assigned Neoplastic Process.

3.2 Inclusion Rules

In this research, the informal, text-based inclusion instructions of the Semantic Network documentation are mapped into precise, implemented inclusion rules. We distinguish between explicit, inherited and implicit inclusion rules. An explicit inclusion instruction is a description of a set of conditions under which it is valid or required for a concept to be assigned two specific semantic types. Explicit inclusion rules are derived from explicit inclusion instructions in the UMLS documentation.

We assign a name to every inclusion rule, for example Anatomical Abnormality with Disease or Syndrome Inclusion Rule. In order to avoid redundant rule names, we always place the two semantic types in a rule name in alphabetical order.

Due to the inheritance of information in the Semantic Network, such a rule may have consequences, going beyond what is expressed by its name. If an explicit inclusion rule is inherited downwards in the Semantic Network, the inherited rule is then referred to as inherited inclusion rule.

For the semantic type Disease or Syndrome, the following usage note proves that the result of inheriting the Anatomical Abnormality with Disease or Syndrome Inclusion Rule is intended: “If an anatomic abnormality has a pathologic manifestation, then it will be given this type as well as a type from the 'Anatomical Abnormality' hierarchy.” (Refer back to Figure 1 to see the hierarchy.) In Table 1, we summarize the three inclusion rules, the numbers of concepts in the intersections of the extents of the semantic types for each rule, and examples of concepts for each rule.

Table 1.

Inclusion rules in the Anatomical Abnormality subhierarchy of the SN

Pair of Semantic Types defining an inclusion rule	Number of Concepts	Example Concepts
(Anatomical Abnormality; Disease or Syndrome)	940	Fistula of Uterus; Dynamic subaortic stenosis
(Congenital Abnormality; Disease or Syndrome)	1,392	Atelocardia; Caroli Disease
(Acquired Abnormality; Disease or Syndrome)	930	Diabetic cataract; Drug-induced peptic ulcer

An implicit inclusion rule cannot be derived from an inclusion instruction in the UMLS documentation. Rather, the fact that an implicit inclusion rule holds for a pair of semantic types needs to be mined from the fact that there are many Metathesaurus concepts assigned exactly this pair of semantic types. It is unlikely that all these assignments are incorrect, and therefore it may be concluded that these two semantic types may occur together. Based on our previous experience with auditing the UMLS for incorrect semantic type assignments [11, 12, 16], a pair of semantic types that has six or more assigned concepts typically defines an implicit inclusion rule.

An interesting case of an inclusion instruction stating inclusion for a whole family of pairs is encountered for semantic types which are descendants of the semantic type Chemical in the Semantic Network. Its definition contains the following instruction: “Almost every chemical concept is assigned at least two types, generally one from the structure hierarchy and at least one from the function hierarchy.” This definition implies a whole “family" of explicit inclusion rules between semantic types in the subhierarchy of Chemical Viewed Structurally and semantic types in the subhierarchy of Chemical Viewed Functionally. Furthermore, the phrase “… and at least one from the function hierarchy” also hints at another interesting family of inclusion rules: A chemical concept may be assigned three semantic types: two from the Chemical Viewed Functionally subhierarchy and one from the Chemical Viewed Structurally subhierarchy.

3.3 Exclusion Rules

There are three categories of exclusion rules corresponding to the above three categories of inclusion rules, and an additional category called redundancy exclusion rules. Explicit exclusion rules are derived from explicit exclusion instructions in the UMLS documentation. Inheritance may spread an explicit exclusion rule of a pair (A; B) of semantic types to all pairs of semantic types (C; D), such that C is a descendant of A and D is a descendant of B in the hierarchy of the Semantic Network. (In this case, children are included among descendants. In addition, A=C or B=D may also hold, but not both.) The results of this inheritance process are inherited exclusion rules. Implicit exclusion rules are defined based on the following reasoning. If there is not a single concept in the over 2.6 million concepts of the UMLS that is assigned a certain pair of semantic types, then it is quite likely that this pair consists of two semantic types that should not occur together, because their combination does not categorize any existing concept in biomedicine. The status of an implicit exclusion rule may change, if such a concept is discovered, but only after an investigation and approval process of a senior UMLS editor, authorizing such a decision.

As for inclusion rules, names are assigned to exclusion rules. Previously, we showed that the text of the usage note of Anatomical Abnormality contained an explicit exclusion instruction, excluding the use of the semantic type Neoplastic Process together with it. The corresponding rule is named the Anatomical Abnormality excluding Neoplastic Process Rule. The semantic types in the rule name are again in alphabetical order. A few interesting exclusion rules of the different categories will be reviewed in the subsections below.

3.3.1 Explicit Exclusion Rules

As an example of an explicit exclusion rule, the children of Finding (Laboratory or Test Result and Sign or Symptom) are mutually exclusive by definition. This implies the Laboratory or Test Result Excluding Sign or Symptom Rule.

In the UMLS documentation it is made explicit that the Anatomical Abnormality Excluding Neoplastic Process Rule also applies to the children of Anatomical Abnormality. (Neoplastic Process has no children). Because of this, there should be no concepts in the Metathesaurus that are simultaneously assigned semantic types from the Anatomical Abnormality subhierarchy and Neoplastic Process. Surprisingly, however, there were a few such concepts in earlier releases of the UMLS, as Table 2 shows for version 2007AC. The last column in Table 2 shows the corrected semantic type assignments for those concepts in both the 2009AA and 2011AA releases of the UMLS.

Table 2.

Two previous violations of Exclusion Rules in the Metathesaurus and their corrections

Illegal Pair of semantic types in 2007AC	Number of Concepts in 2007	Concepts with Illegal Assignments	Corrected semantic type Assignment of Concept in the UMLS in 2009AA and 2011AA
(Anatomical Abnormality; Neoplastic Process)	1	Acquired arteriovenous aneurysm	Pathologic Function
(Congenital Abnormality; Neoplastic Process)	1	Congenital melanocytic nevus	Neoplastic Process

3.3.2 Inherited Exclusion Rules

Examples of inherited exclusion rules will be discussed in the Results Section, in Subsection 4.2.2.

3.3.3 Redundancy Exclusion Rules

According to the instructions of the National Library of Medicine, redundant assignments of semantic types are prohibited [20] in the UMLS. In other words, if one semantic type is assigned to a concept, then the parent and (if they exist) ancestors of this semantic type may not be assigned to this concept. Thus, it is possible to create a list of pairs of a semantic type and each of its ancestors (including the parent). Every element in this list defines an exclusion rule.

For example, the semantic type Neoplastic Process has the parent Disease or Syndrome. Its non-parent ancestors are Pathologic Function, Biologic Function, Natural Phenomenon or Process, Phenomenon or Process and Event. Thus the pairs (Neoplastic Process; Disease or Syndrome), (Neoplastic Process; Pathologic Function), (Neoplastic Process; Biologic Function), etc. are prohibited combinations. Each of these pairs defines a redundancy exclusion rule.

The Semantic Network contains 88 leaf semantic types, i.e., semantic types without children. Each leaf defines a unique path, starting at the leaf and ending at one of the two roots, Entity or Event. We define that the root nodes of the Semantic Network are at level zero. If we define that each child of a node at level m is considered to be at level m+1, we can assign a level number to every node in the Semantic Network. Furthermore, a path from a node A at level m to its root will contain m nodes (excluding A itself). This numbering is convenient and is the reason for the choice that the root is assigned the level 0 instead of 1.

Under these assumptions, a semantic type at level m excludes all the m semantic type(s) above it. This holds true for leaf nodes and for non-leaf nodes. Thus, to compute the total number of prohibited pairs of semantic types, the distribution of semantic types over levels is needed. Given that the Semantic Network has semantic types at levels 0 to 7, the total number of prohibited pairs (PP) can be computed as the product of the number S(m) of semantic types at a level m with the level number (m), summed over all levels.

$$PP={\displaystyle {\sum }_{m=\mathrm{1..7}}m*S(m)}$$ (1)

3.3.4 Implicit Exclusion Rules

When given n elements, there are n*(n-1)/2 ways to choose a pair out of these n elements, assuming that pairs are order independent, and an element cannot form a pair with itself. Hence there are potentially 133*(133-1)/2 = 8778 pairs of semantic types. Out of this total of 8778 distinct semantic type pairs, there are only 199 pairs for which concepts have been assigned this combination of two semantic types in the UMLS.

If a pair of semantic types is not assigned to any concept, i.e., the intersection of their extents is empty, then one should wonder whether this pair should be defined as exclusive. However, with 8579 (=8778−199) candidate pairs such an investigation is difficult. For some of these pairs we have exclusion rules of the other categories discussed earlier. But those amount only to a small fraction of the 8579 possibilities.

For the remainder of this investigation we adopt the following pragmatic attitude towards this issue. A pair of semantic types that is not assigned to any concept is assumed to define an implicit exclusion rule. This is similar to the closed world assumption in logic programming, which states that if a fact is not explicitly known, it is assumed not to hold (negation as failure) [21].

3.4 Implementation of the Inclusion and Exclusion Rules in a Computer System

We have developed an algorithm adviseEditor that is passed two or more semantic types as input and returns the rule-category that applies to these semantic types. For reasons of exposition, we will start with the description and the algorithm for the simplest case where only two semantic types are assigned to a concept. At the end of this section we will describe how the system is extended to handle cases where a concept is assigned more than two semantic types.

Redundancy exclusion is the result of a pair of semantic types standing in an ancestor/descendant (or parent/child) relationship in the Semantic Network. Thus, the test for this case is expressed in the algorithm below by ((S1 is an ancestor of S2) OR (S2 is an ancestor of S1)). For the purpose of the algorithm, we treat parents as ancestors. Explicit inclusion rules and explicit exclusion rules cannot be found algorithmically at the current state-of-the-art, as they are based on natural language descriptions in the UMLS documentation. Thus, the list of pairs (S1; S2) and their mirror images (S2; S1) that fall into the explicit inclusion and explicit exclusion rule-categories were found by manual research and then prestored in two arrays of semantic type pairs, called Explicit_Inclusions_Array and Explicit_Exclusions_Array.

Cases of inclusion and exclusion that are based on inheritance are processed by looking upward in the Semantic Network, with the purpose of finding semantic types that are parents or ancestors that could be the source of inheritance of a specific inclusion or exclusion rule. Thus they do not need to be prestored.

We note that some pairs of semantic types may be categorized in contradictory ways, due to different rules. For example the pair (Anatomical Abnormality; Neoplastic Process) is explicitly excluded in the UN of the semantic type Anatomical Abnormality. However, the same pair may also be categorized by an inherited inclusion rule, since the pair (Anatomical Abnormality; Disease or Syndrome) is categorized with an explicit inclusion rule, due to a remark in the UN of Anatomical Abnormality about concepts that should be also assigned Disease or Syndrome, and because Disease or Syndrome is the parent of Neoplastic Process. A similar contradiction may also occur between an explicit exclusion rule and cases of implicit inclusion or “more research required.” In all such cases, the explicit rule (either inclusion or exclusion) should override the other kinds of rules. In the algorithm below this preference is implemented by checking for explicit inclusion and explicit exclusion before checking for other options such as inheritance. The symbol ∈ is read as “is in.” Two vertical bars | | define the number of elements of the set in between them.

Algorithm adviseEditor(S1 SemanticType, S2 SemanticType) {

if (S1= S2)

{return ‘Input not valid’}

if ((S1 is an ancestor of S2) OR (S2 is an ancestor of S1))

{return ‘Prohibited by Redundancy Exclusion’}

else if (S1, S2) ∈ Explicit_Inclusions_Array

{return ‘Permitted by Explicit Inclusion’}

else if (S1, S2) ∈ Explicit_Exclusions_Array

{return ‘Prohibited by Explicit Exclusion’}

else if (any_ancestor(S1), any_ancestor(S2)) ∈ Explicit_Inclusions_Array

{return ‘Permitted by Inherited Inclusion’}

else if (any_ancestor(S1), any_ancestor(S2)) ∈ Explicit_Exclusions_Array

{return ‘Prohibited by Inherited Exclusion’ }

else if (|Extent(S1) ∩ Extent(S2)| >= 6)

{return ‘Most likely Permitted by Implicit Inclusion’}

else if (|Extent(S1) ∩ Extent(S2)| = 0)

{return ‘Most likely Prohibited by Implicit Exclusion’}

else if (|Extent(S1) ∩ Extent(S2)| is between 1 and 5)

{return ‘More Research Required.

Check all Concepts that are assigned both S1 and S2.

If at least one is simultaneously, correctly assigned S1 and S2, this pair is Permitted by Implicit Inclusion.

If they are all wrongly assigned either S1 or S2 or both, this pair is Prohibited by Implicit Exclusion.' }

}

This algorithm is a concise summary of the computer implementation described in the Results Section. However, a lookup table was utilized to accelerate the performance of the adviseEditor system. For example, the line |Extent(S1) ∩ Extent(S2)| >= 6 requires a multi-step computation. The two vertical bars | | indicate that the number of elements of the set between them is returned. Similarly, the line (any_ancestor(S1), any_ancestor(S2)) ∈ Explicit_Inclusions_Array requires an extensive computation. Such results were stored in a lookup table. The algorithmic notation hides these complications from the reader.

The adviseEditor algorithm was executed for every pair of distinct semantic types from the Semantic Network, and the rule-category for each pair was recorded. The total number of occurrences of each rule-category was then computed. These numbers will be reported as results. While testing the algorithm, contradictions between rule-category assignments and actual concept assignments in the Metathesaurus were found. These contradictions will be reported in the Results Section.

Next, we consider cases where a concept is assigned more than two semantic types. We start with the case of a concept assigned three semantic types. The cases of more semantic types will be handled similarly, as will be explained later.

Let S1, S2 and S3 be the three semantic types assigned to a concept C. We refer to (S1; S2; S3) as a triple of semantic types. In the documentation of the UMLS the possibility of an exclusion rule for three or more semantic types is not mentioned. However a triple (S1; S2; S3) is excluded if any of the three pairs (S1; S2), (S1; S3) or (S2; S3) is excluded. Hence, when considering a triple (S1; S2; S3) adviseEditor will test each of the three pairs for explicit exclusion, inherited exclusion and redundancy exclusion. If any of these rules holds for any of the three pairs, the triple is also excluded according to the most stringent rule-category of all the excluded pairs. (In this context redundancy exclusion is more stringent than explicit exclusion, which in turn is more stringent than inherited exclusion.)

With regard to inclusion rules for triples the situation is different. The definition of Chemical contains the following instruction: “Almost every chemical concept is assigned at least two types, generally one from the structure hierarchy and at least one from the function hierarchy” (see Section 3.1). This implies the possibility of an inclusion rule for triples (S1; S2; S3) where S1 is a descendant of Chemical Viewed Structurally and S2 and S3 are descendants of Chemical Viewed Functionally. Such assignments of three semantic types occur only in the subtree rooted at Chemical. No other possibility of an inclusion rule for three or more semantic types is mentioned, which eliminates explicit inclusion and inherited inclusion rules for triples, unless one semantic type is a descendant of Chemical Viewed Structurally and two are descendants of Chemical Viewed Functionally.

What about other kinds of triples? If any of the three semantic types is not a descendant of Chemical then the triple is categorized as implicit exclusion, since there are no concepts with such triples in the UMLS. All concepts assigned more than two semantic types are chemical concepts.

Next we discuss cases of three descendants of Chemical that do not follow the pattern of the above inclusion rule, e.g., there could be two structural and one functional, semantic type. For such triples, we first test their three pairs for explicit, inherited or redundancy exclusion as described above. If no pair is excluded we handle these triples, just like pairs of semantic types. If a triple is assigned to more than five concepts, it defines an implicit inclusion rule. If no concept is assigned such a triple, it defines an implicit exclusion rule. Finally, if a triple is assigned to between one and five concepts, its status will be “more research required.” There are only 178 triples of semantic types assigned to concepts. Most of them follow the pattern of one structural and two functional semantic types of the above explicit inclusion rule. The few remaining triples are stored in a lookup table where they are listed with corresponding numbers of concepts, allowing fast processing.

An interesting research issue arose out of the fact that sometimes a quadruple (4) or quintuple (5) of semantic types is assigned to one or more concepts. If the combination of four semantic types is allowed, then any three of those four (or five) must also be allowed together.

For the quadruple case there are four different possibilities to choose three semantic types from them. For the quintuple case, the number of ways to choose three out of five is computed by: 5*4 / (5−3)! = 20/2 = 10 possibilities. There are only 31 quadruples of semantic types assigned to concepts in the UMLS. Furthermore, only triples that do not follow the pattern of one structural and two functional semantic types need to be considered. The number of triples added to the lookup table in this way is quite limited, since most of these triple are already in the lookup table, due to their independent existence as triples of semantic types assigned to concepts.

For example, for the quadruple (Amino Acid, Peptide, or Protein; Pharmacologic Substance; Immunologic Factor; Indicator, Reagent, or Diagnostic Aid) assigned to 146 concepts, only one triple consisting of the last three functional semantic types needs to be considered. But this triple already appears independently in the UMLS, assigned to 94 concepts.

The only quintuple in the UMLS, (Amino Acid, Peptide, or Protein; Pharmacologic Substance; Biologically Active Substance; Indicator, Reagent, or Diagnostic Aid; Hazardous or Poisonous Substance) is assigned to only one concept 131I-TM-601. The adviseEditor system categorizes this quintuple as "Explicit Exclusion," because one of its pairs Pharmacologic Substance and Hazardous or Poisonous Substance is categorized as "Explicit Exclusion." In other words, there is not a single valid quintuple in the UMLS, and therefore no triples derived from a quintuple were added to the lookup table.

The details of processing the quadruples are analogous to the treatment of those triples that do not follow the above mentioned explicit inclusion rule for triples. For brevity, we do not discuss these details. Since there are currently no cases of six semantic types assigned to a concept (for the whole UMLS), such a case is not incorporated into the adviseEditor system. The implementation of the procedure for handling between three and five semantic types was a straightforward extension of the code for pairs, and therefore no code is provided.

3.5 Evaluation of the adviseEditor system

The adviseEditor system is only needed for UMLS concepts assigned more than one semantic type. In order to evaluate the effectiveness of the adviseEditor system, we generated a sample of concepts as follows. We selected pairs of non-chemical semantic types such that there is at least one and there are at most five concepts with those pairs assigned. This sample was processed with the adviseEditor system. The sample concepts were also reviewed by a human auditor. These review results were used to evaluate the performance of the adviseEditor system.

This choice of concepts for our sample is based on the fact that we consider combinations of semantic types assigned to just a few concepts as problematic. Such combinations of semantic types will be assigned “more research required” by adviseEditor. Those are the kinds of concepts where the adviseEditor system is more likely to fail and needs to be tested. In contrast, we expect the system to perform relatively better for combinations of semantic types assigned to many concepts, such as for example the 658 concepts assigned the semantic types Vitamin and Pharmacologic Substance.

The problematic nature of the former kind of combinations is expressed by the fact that the “more research required” result is returned, by the adviseEditor system only after all the other choices have been tested. Thus, even though a concept with two assigned semantic types may fulfill the conditions of “more research required,” the two semantic types may also fulfill more stringent conditions, such as explicit exclusion. Indeed, this was found to be the case for several concepts in this sample, as will be described in Section 4.7.

4 Results

4.1 Inclusion Rules for Chemical Semantic Types

For brevity, we are not covering all inclusion rules but concentrate on two especially interesting cases.

4.1.1 Inclusion Rules between Chemical Viewed Structurally & Chemical Viewed Functionally Semantic Types

As explained in Section 3.1, there is a family of explicit inclusion rules where the first semantic type is a descendant of Chemical Viewed Structurally and the second is a descendant of Chemical Viewed Functionally. There are 10 descendants of Chemical Viewed Structurally and 20 of Chemical Viewed Functionally. Hence, the total number of explicit inclusion rules for this family is 10 × 12 = 120. For example, there are 82,059 concepts assigned the pair (Organic Chemical; Pharmacologic Substance).

4.1.2 Pairs of Chemicals Viewed Functionally Inclusion Rules

As explained in Section 3.1, there is a family of explicit inclusion rules where both semantic types are descendants of Chemical Viewed Functionally. Chemical Viewed Functionally has 12 descendants. The total number of potential explicit inclusion rules in this case is (12 * 11)/2 = 66. Table 3 shows the numbers of concepts in intersections of descendants of Chemical Viewed Functionally with each other. Column headers are identical to row names and are abbreviated as needed. The children of Pharmacologic Substance and Biologically Active Substance are listed following them, respectively. The first column in Table 3 shows that Pharmacologic Substance has intersections with large extents with most other semantic types in the Chemical Viewed Functionally subhierarchy. The only empty intersection is with Receptor.

Table 3.

Intersections of descendants of Chemical Viewed Functionally with each other

--	Pharm. Subst.	Antib.	Biomed. or Dental Material	Biologically Active Substance	Neuroreactive Subst. or Biog. Amine	Horm.	Enz.	Vitam.	Immunol. Factor	Recep.	Indicat., Reagent or Diag. Aid	Hazard. or Poisonous Substance
Pharmacologic Substance	--	--	--	--	--	--	--	--	--	--	--	--
Antibiotic	Redun- dant	--	--	--	--	--	--	--	--	--	--	--
Biomedical or Dental Material	158	0	--	--	--	--	--	--	--	--	--	--
Biologically Active Substance	803	17	3	--	--	--	--	--	--	--	--	--
Neuroreactive Substance or Biogenic Amine	9	0	0	Redundant	--	--	--	--	--	--	--	--
Hormone	96	0	0	Redundant	12	--	--	--	--	--	--	--
Enzyme	93	0	0	Redundant	0	0	--	--	--	--	--	--
Vitamin	658	0	2	Redundant	0	0	0	--	--	--	--	--
Immunologic Factor	2234	0	0	Redundant	0	0	1	0	--	--	--	--
Receptor	0	0	0	Redundant	0	1	3	0	12	--	--	--
Indicator, Reagent or Diagnostic Aid	479	5	16	3	0	0	1	0	137	0	--	--
Hazardous or Poisonous Substance	97	0	1	498	0	0	10	0	9	0	3	--

The intersection of Pharmacologic Substance with Antibiotic in Table 3 is marked “redundant,” since the assignment of Antibiotic to a concept makes the assignment of Pharmacologic Substance to this concept redundant (see Section 3.2.3). Out of 66 pairs of semantic types, only 27 are actually assigned to concepts. The difference between the 66 explicit inclusion rules and the 27 non-empty intersections reinforces the fact that explicit inclusion rules enable a combination of semantic types, but the option is not always materialized.

The same observation holds true for the family of inclusion rules in Section 4.1.1. For some of the 120 rules there are currently no concepts. For example, the pair (Receptor; Organic Chemical) is not assigned to any concept.

Figure 3 shows a three dimensional view of a matrix consisting of intersections of extents of pairs of semantic types from the Chemical Viewed Functionally subhierarchy. The number of concepts in an intersection is expressed by the height of the corresponding bar. In order to better differentiate the heights of the bars, a logarithmically scaled z axis is used.

Figure 3.

Intersections of pairs of functional chemical semantic types

As can be seen in Figure 3, Pharmacologic Substance has intersections with large extents with most other semantic types in the Chemical Viewed Functionally subhierarchy (see second row of bars in Figure 3, starting from the front).

We note that this figure is symmetrical, having the same set of semantic types on the x and the y axes. There are no bars in the diagonal (meaningless pairs of a semantic type with itself). However, we are displaying each pair of semantic types at both possible locations, to simplify the mental retrieval from this three-dimensional view, since by following the horizontal color coding, one can easily see all intersections of a given semantic type. The total number of potential bars in Figure 3 is (12 * 12 − 12) = 132. The difference between the 132 potential bars and the 54 visible bars constitutes another way of visualizing the fact that possible pairs of semantic types are not always materialized.

4.2 Exclusion Rules Results

For brevity we are not reporting an exhaustive list of exclusion rules, but concentrate on interesting and typical cases.

4.2.1 Explicit Exclusion Rules

The UN of the semantic type Finding contains the instruction that “Only in rare circumstances will findings be double-typed with either 'Pathologic Function' or 'Anatomical Abnormality’.” We interpret this usage note to imply two explicit exclusion rules, the Finding Excluding Pathologic Function Rule and the Anatomical Abnormality Excluding Finding Rule.

For the semantic type Activity the UN contains the instruction “In general, concepts will not receive a type from both the 'Activity' and the 'Behavior' hierarchies.” This expresses the Activity Excluding Behavior Rule.

The definition of Organophosphorus Compound contains the instruction that “Excluded are phospholipids, sugar phosphates, phosphoproteins, nucleotides, and nucleic acids.” This implies four exclusion rules, which are the Lipid Excluding Organophosphorus Rule, the Amino Acid, Peptide or Protein Excluding Organophosphorus Rule the Carbohydrate Excluding Organophosphorus Rule and the Nucleic Acid, Nucleoside, or Nucleotide Excluding Organophosphorus Rule.

Table 4 lists eleven pairs of semantic types for which an explicit exclusion rule exists, nevertheless, concepts have been assigned to those pairs. The number of problematic concepts for each exclusion rule is listed in Column 2 and a sample concept is listed in Column 3. All 278 concepts referred to in Table 4 have a wrong semantic type assignment, according to an explicit exclusion rule.

Table 4.

Eleven pairs prohibited by explicit exclusion, with concept assignments

Pairs of semantic types defining an explicit exclusion rule	# of Conc.	Example concept
(Medical Device; Research Device)	12	C0600364 Biosensors
(Nucleic Acid, Nucleoside, or Nucleotide; Organophosphorus Compound)	25	C0674527 5'-O-phosphonylmethylthymidine
(Hazardous or Poisonous Substance; Pharmacologic Substance)	97	C0145114 teleocidin B
(Element, Ion, or Isotope; Inorganic Chemical)	10	C2347051 Mn2+
(Amino Acid, Peptide, or Protein; Organophosphorus Compound)	46	C0064331 keyhole limpet hemocyanin phosphonamidate conjugate
(Carbohydrate; Organophosphorus Compound)	46	C0063569 inositol 1,4,5-triphosphorothioate
(Lipid; Organophosphorus Compound)	35	C0256611 EPC-NPH
(Body Substance; Pharmacologic Substance)	1	C1976001 Blood product units & Blood product unit
(Organic Chemical; Inorganic Chemical)	1	C2975881 Ringerfundin
(Finding; Pathologic Function)	2	C0267995 Fluid volume disorder
(Organic Chemical; Element, Ion, or Isotope)	3	C0302933 Natural graphite
TOTAL	278

The semantic type Clinical Drug has a UN with the instruction “Do not double type with Pharmacologic Substance, Antibiotic or other chemical semantic types.” This defines yet another family of explicit exclusion rules.

4.2.2 Inherited Exclusion Rules

If Finding excludes Pathologic Function (see above), then, by inheritance of explicit exclusion rules, Finding should also exclude the descendants of Pathologic Function, such as Disease or Syndrome. In version 2007AC, many concepts contradicting such exclusion rules existed. These were corrected in version 2009AA. In that version, Finding did not have any concepts with a second semantic type assigned to them.

However, in version 2011AA, Finding and Pathologic Function were assigned to two concepts, in spite of the explicit exclusion rule. Furthermore, Finding and Disease or Syndrome are both assigned to three concepts, in contradiction to inherited exclusion. In addition, Finding is assigned to other groups of concepts that are assigned additional semantic types in contradiction to exclusion rules, as follows: Finding and Sign or Symptom (1 concept) (redundancy exclusion), Finding and Acquired Abnormality (1) (inherited exclusion), and Finding and Congenital Abnormality (2) (inherited exclusion). In total, there are nine new assignments that have been introduced into the UMLS for Finding, between version 2009AA and version 2011AA, that are likely to be erroneous. For example, in version 2011AA, E(Finding) ∩ E(Acquired Abnormality) contains the concept Flexion contracture of proximal interphalangeal joint.

In summary, a set of errors was corrected between 2007 and 2009 and then new errors violating these rule-categories were introduced by 2011. This indicates the importance for consulting the adviseEditor system before assigning a pair of semantic types to a new concept.

4.2.3 Redundancy Exclusion Rules

As noted in Section 3.2.3, there are 88 leaves in the two trees in the Semantic Network. Every one of these leaves defines a path to its respective root. In total, there are 2 semantic types at level 0, 4 are at level 1, 20 at level 2, 40 at level 3, 24 at level 4, 19 at level 5, 21 at level 6 and 3 at level 7.

Using formula (1) from Section 3, with 4 * 1 + 20 * 2 + 40 * 3 + 24 * 4 + 19 * 5 + 21 * 6 + 3 * 7 we get exactly 502 redundancy exclusion rules, which correspond to about 5.7% of the 8778 pairs of semantic types. This result is in agreement with the result found by our program.

4.3 The Rule-Category “More Research Required”

Our previous research shows that when there are six or more concepts assigned a pair of semantic types, unless appearing as an explicit exclusion rule or inherited exclusion rule, one can safely assume an implicit inclusion rule [17]. Similarly, one can safely assume an implicit exclusion rule when there are no concepts assigned a pair of semantic types. However, what happens when between one and five concepts have been assigned a specific pair of semantic types?

In such a case, the UMLS editor will need to investigate all those concepts, whether the assignment of these two semantic types is really justified. If all such concepts are modified such that they do not have this pair of semantic types assigned, then the pair will be converted into a case of implicit exclusion. In that case, no new concepts may be assigned this pair of semantic types. On the other hand, if the assignment of these two semantic types is justified for an existing concept, this pair should be transitioned to the status of implicit inclusion rule and may also be assigned to a new concept.

In the 2011AA version of the UMLS, we have found 30 pairs of semantic types assigned the rule-category “more research required.” A detailed analysis of these cases goes beyond the scope of this paper.

4.4 Numbers of semantic type pairs in each rule-category

Table 5 shows the numbers of pairs of semantic types (S1; S2) assigned to each rule-category. The results in rows 1 to 8 follow exactly the order in which the corresponding tests are performed in the algorithm adviseEditor. The pairs (S1; S2) and (S2; S1) are only counted once.

Table 5.

Numbers of semantic type pairs in each rule-category

Row #	Rule Category	Number of Occurrences
1	Redundancy Exclusion	502
2	Explicit Inclusion	181
3	Explicit Exclusion	104
4	Inherited Inclusion	30
5	Inherited Exclusion	71
6	Implicit Inclusion	34
7	Implicit Exclusion	7826
8	More Research Required	30

4.5 Visualizing the Space of Semantic Type Pairs

While we have concentrated on an algorithmic treatment of inclusion and exclusion rules, the question naturally arises whether pairs of semantic types could not be displayed as a two-dimensional matrix. Displaying a matrix with 8778 numerical values on 8.5″ by 11″ paper is impossible. However, we have attempted to create a diagram approximating such a display using color coding.

Figure 4 shows color-coded rule-categories for pairs of semantic types. The 133 semantic types are numbered by the NLM from T001 to T203 (there are gaps). Every point encodes the pair of semantic types defined by its values on the x and y axes. The diagonal through the origin (T001, T001) defines pairs of identical semantic types.

Figure 4.

Color-coded rule-categories for pairs of semantic types

The semantic type Entity (T071) naturally is excluded by the largest number of other semantic types due to redundancy exclusion, as it is the root of the larger of the two trees of the Semantic Network. Thus, the longest orange lines in the diagram are at the row and column of T071. Other long lines are at T051, which correspond to Event, the other root of the Semantic Network. Together, these two semantic types are excluded by every other semantic type, except by each other. Thus, the lines at T071 and T051 cover almost the complete x dimension and y dimension of the diagram.

In Figure 4, we see an area of red, marking explicit inclusion, above and to the right of T103 (Chemical). This illustrates the inclusion rules among the Chemical Viewed Functionally semantic types, discussed in Section 4.1.2 and between the Chemical Viewed Functionally and the Chemical Viewed Structurally semantic types, discussed in Section 4.1.1.

4.6 Implementation of the adviseEditor system

The Web-based adviseEditor system was developed to help the editors and auditors of the UMLS to determine which combinations of semantic types are permitted and which are prohibited for a new concept. Processing can be done for single concepts or in batch mode for large groups of concepts. A user can enter two, three, four or five semantic types for single concepts. The Batch Processing Utility can handle a series of concepts, each assigned a combination of between two and five semantic types. The adviseEditor system is accessible at http://nat.njit.edu/NATServlet.

In Figure 5, a sample result is shown, as returned by the Batch Processing Utility. Results are sorted according to the number of semantic types in the input combination. The returned results may be saved in a file for future use.

Figure 5.

Example result returned by the Batch Processing Utility of adviseEditor

In the interactive utility for three semantic types in Figure 6, the user may choose three semantic types from three drop down menus. After the user clicks the button “Submit,” the rule-category of the selected triple will be shown. The interactive utilities for two, four and five semantic types appear and work similarly. In an experiment, the average interactive processing time for two semantic types was found to be 453 milliseconds.

Figure 6.

Interface of Interactive Utility for three Semantic Types

4.7 Evaluation Study for the performance of the adviseEditor system

In order to evaluate the performance of the adviseEditor system, we generated a sample of concepts as follows. We determined all pairs of non-chemical semantic types in the 2011AA UMLS release, such that there is at least one and there are at most five concepts with those pairs assigned. There are only 32 such pairs in the release. We then selected all 65 concepts assigned any one of these 32 pairs of semantic types and processed the sample with the adviseEditor system. These 65 concepts were also reviewed by a human auditor, one of the authors (JX), trained in both medicine and medical terminologies. Our auditor is not an expert in chemistry, thus the study was limited to the non-chemical combinations. Naturally, our auditor was not given access to the adviseEditor system.

Among the 32 pairs of semantic types audited, the 16 pairs listed in Table 6 are new in the 2011AA version of the UMLS. The column Rule-Category indicates which category the pair of semantic types in this row belongs to. The column #cpts contains the number of concepts that are assigned this pair of semantic types. Notably, the column Rule-Category indicates a kind of exclusion rule for every pair in Table 6, and what kind of exclusion rule it is. Thus, the column #cpts (number of concepts) should ideally contain 0 in every row.

Table 6.

New pairs of non-chemical semantic types with few (1 to 5) concepts in 2011AA. All values in the column #cpts (number of concepts) should ideally be 0.

Line	Semantic Type A	Semantic Type B	Rule category	# cpts	Appeared in prev. UMLS release?
1	Acquired Abnormality	Finding	Inherited Exclusion	1	2007AC
2	Body Part, Organ, or Organ Component	Substance	Implicit Exclusion	1	No
3	Body Substance	Pharmacologic Substance	Explicit Exclusion	1	No
4	Congenital Abnormality	Finding	Inherited Exclusion	2	2007AC
5**	Clinical Attribute	Finding	Implicit Exclusion	2	2007AC
6	Diagnostic Procedure	Finding	Implicit Exclusion	1	No
7	Disease or Syndrome	Finding	Inherited Exclusion	3	2008AB
8**	Finding	Health Care Activity	Implicit Exclusion	1	2008AA
9	Finding	Injury or Poisoning	Implicit Exclusion	2	No
10	Finding	Pathologic Function	Explicit Exclusion	2	2007AC
11	Finding	Sign or Symptom	Redun- dancy Exclusion	1	No
12	Population Group	Mental or Behavioral Dysfunction	Implicit Exclusion	1	No
13**	Pharmacologic Substance	Plant	Implicit Exclusion	1	2008AA
14**	Functional Concept	Spatial Concept	Implicit Exclusion	1	2008AA
15**	Functional Concept	Therapeutic or Preventive Procedure	Implicit Exclusion	1	2007AC
16	Bacterium	Virus	Implicit Exclusion	1	No

The last column, “Appeared in previous UMLS release?” shows whether and when a pair appeared in a previous UMLS release prior to 2010AB, before it disappeared subsequently due to auditing efforts, and (re)appeared in the 2011AA release. Nine out of the 16 pairs appeared in the past, according to our research, covering the period from 2006AC to 2010AB.

For six of the 16 rows in Table 6, using the adviseEditor system would have warned the UMLS editors about introducing erroneous pairs of semantic types for new concepts, because these pairs contradict explicit exclusion, inherited exclusion or redundancy exclusion. For example, Finding and Pathologic Function, a case of explicit exclusion, are assigned to Fluid volume disorder. Our auditor suggested assigning Sign or Symptom instead. Congenital Abnormality and Finding with the category inherited exclusion, are assigned to Labial hypoplasia. Finding was considered a wrong assignment by our auditor. Finding and Sign or Symptom with the category redundancy exclusion, are assigned to the concept Subungual swelling. The redundant assignment of Finding was deemed to be wrong by our auditor.

The other ten of the 16 rows in Table 6 are cases of “implicit exclusion.” The entries for these rows assume that adviseEditor would have been applied before the first concept was assigned such a pair when creating the UMLS 2011AA release. However, after creating the UMLS 2011AA release the system would have returned “more research required” instead, since in this release such semantic type pairs were already assigned to one or a few concepts (according to the column #cpts).

For the purpose of evaluating the adviseEditor system, we need to assume that the UMLS editors would have used it when preparing the UMLS 2011AA release. When the very first assignment of each one of the ten pairs of semantic types to a concept was attempted, “implicit exclusion” would have been the result of adviseEditor, which is what appears in Table 6. This assignment would only be allowed with an extra level of approval by a senior editor or a team of editors (as will be suggested in the Discussion Section). As we shall see, our auditor would have approved only a few of those pairs, preventing the creation of wrong semantic type assignments. Whenever such a pair would have been approved for one concept, the result of adviseEditor would have changed to “more research required” for this pair, because the UMLS would have this pair assigned to a concept at that point in time. If an auditor presents several concepts with the same pair of semantic types (prohibited by implicit exclusion) for approval, then all these concepts will need to be evaluated by the supervisor or team.

Indeed, looking back at Table 6, there were six concepts assigned five new pairs of semantic types marked “implicit exclusion,” which had appeared in a previous release, but were removed after an audit. (The line numbers of those five pairs are marked by “**”.) Considering the fact that only two of these five pairs were accepted by our auditor as correct, namely (Pharmacologic Substance; Plant) and (Functional Concept; Spatial Concept), there is a high likelihood that approvals would not have been given by the UMLS editors for the other “**” cases either.

Table 7, shows in the first row that 3, 8, 1 and 12 concepts, respectively, were categorized by adviseEditor as explicit exclusion, inherited exclusion, redundancy exclusion or implicit exclusion. That is, for these 24 concepts, the assigned pairs were deemed wrong by adviseEditor. Our auditor agreed with 19 (79%) of these recommendations of the system. We note that our auditor missed one case of explicit exclusion for the concept Blood product units | Blood product unit assigned Body Substance and Pharmacologic Substance.

Table 7.

Results of adviseEditor system and auditor’s evaluation of the results of the adviseEditor system

	Explicit Ex- clusion	Inherited Ex- clusion	Redundan -cy Exclusion	Implicit Ex- clusion	More Research Required	Total
# of concepts categorized by adviseEditor	3	8	1	12	41	65
# of concepts confirmed by auditor	2	8	1	8	28	47
# of concepts not confirmed by auditor	1	0	0	4	13	18

For “more research required” the issue is different. In this case the auditor agrees with adviseEditor whenever s/he considers the pair as acceptable, because there is already a concept with this assignment in the UMLS. It is important to understand that this is an evaluation of the adviseEditor system, and not an evaluation of the UMLS. Thus"more research required” does not mean that the auditor needs to go and check those previous assignments. As indicated in Table 7, 68% of the 41 concepts (28/41) categorized by adviseEditor as “more research required” were confirmed by the auditor.

Based on Table 7, we calculated the performance of the adviseEditor system for the given sample. The calculation used the determination of the auditor as a gold standard.

The accuracy (the proportion of the assessments of the system which are confirmed by the auditor) is (2 + 8 + 1 + 8 + 28)/65 = 47/65 = 0.72. The precision (the ratio of the semantic type assignments reported as correct by the system, as confirmed by the auditor, to all concepts reported as correct by the system) is 28/41 = 0.68. The recall (the ratio of semantic type assignments reported as correct by the system, as confirmed by the auditor, to all correct concepts) is 28/(28 + 1 + 4) = 28/33 = 0.85. The F-measure (harmonic mean) is F = 2 * Recall * Precision / (Recall + Precision) = 2 * 0.85 * 0.68 / (0.85+0.68) = 0.76.

The sample used in this study is too small to establish statistical significance. However, the size of this sample could not be increased, because we already included all 65 relevant concepts from the UMLS 2011AA release in it, as explained in Section 3.7.

5 Discussion

It is interesting to note the ratio of explicit versus inherited rules, namely, 181:30 for inclusion rules and 104:71 for exclusion rules, according to Table 5. Intuitively, one would expect the number of inherited rules to be larger than the number of explicit rules. The reason for that is that if an explicit rule is stated between the semantic types X and Y, and if X has m descendants and Y has n descendants, then there may be m*n inherited rules between descendants of X and Y.

However, the reality is different. One reason for that is that many explicit rules are stated between semantic types that are leaves in SN, or between semantic types with just one or two descendants. The potential exceptions regarding descendants of Chemical Viewed Functionally or between them and descendants of Chemical Viewed Structurally are not listed as inherited, since explicit rules are given in the documentation for these two subhierarchies.

An interesting observation from Figure 4 is that areas of inherited exclusion (blue) appear adjacent to areas of explicit exclusion (purple). A similar observation can be made for the corresponding inclusion rules (appearing as green and red). The interpretation of this observation is that the semantic types for which inherited rules hold typically appear after (in the UMLS numbering scheme) the semantic types for which the explicit rules are stated.

For some implicit exclusion rules it is surprising that they were not made explicit. For example, the UMLS/SN definition for Fish is: “A cold-blooded aquatic vertebrate characterized by fins and breathing by gills. Included here are fish having either a bony skeleton, such as a perch, or a cartilaginous skeleton, such as a shark, or those lacking a jaw, such as a lamprey or hagfish.”

The Linnaean system of classification for animals assumes the exclusiveness of parallel branches. The above definition does not state that fish and mammals are considered exclusive in the animal kingdom tree. Therefore, the Fish Excluding Mammal Rule cannot be discerned from the Semantic Network itself. We observe that this is a case of specialization of a parent semantic type into several children in the Semantic Network, done with the intention that the extents of all sibling semantic types should be disjoint. In other words, being a sibling implies the existence of an exclusion rule. This pattern is repeated in the taxonomy of life forms. For the semantic types Vertebrate, Animal (and Organism) we know from the animal kingdom categories that their children are exclusive.

If any concepts were to have two assignments of semantic types from parallel branches of the part of the Semantic Network that mimics the animal kingdom categorization, then this would be a serious error. In version 2007AC there was one such pair. The two semantic types Invertebrate and Alga were assigned to 19 concepts, e.g., Euglena, Plankton, and Discoplastis spathirhyncha. This violation has been corrected. Subsequently, these two semantic types were removed from the Semantic Network, and thus no concepts can have those assignments in 2011AA.

Around 2009, the NLM implemented an automatic quality assurance procedure which removes redundant semantic type assignments before each release of the UMLS [20]. Hence, there are in general no more illegal semantic type pairs due to redundant assignments in the UMLS, although, adviseEditor, exposed one case (see Table 6).

Our evaluation showed a relatively high performance of the adviseEditor system, exposing many semantic type assignments in contradiction to UMLS rules. We noted in Section 4.7 that the reference standard used was not perfect, but this is not unusual when dealing with human decisions about complex choices.

We propose the use of the described adviseEditor system as a mechanism to support the process of assigning semantic types to new concepts added to the UMLS or updated due to integration of a new release of a source terminology. This system can inform UMLS editors concerning whether a specific combination of semantic types is permitted or prohibited, rather than considering the assignment of one semantic type in isolation from other existing assignments. The use of the adviseEditor system, categorizing a pair of semantic types as permitted, prohibited, etc., is expected to prevent insertions of new erroneous semantic type assignments, and also to expedite the editors’ work. Considering the shortage of human expert editors for terminologies in general and for the UMLS in particular, expediting the editorial process will free up editors to work on other relevant tasks.

Should the situation arise that a new concept is assigned a pair of semantic types from the implicit exclusion rule-category, then this assignment and the concept itself need to be carefully investigated to determine whether they are valid. We propose a policy that no “ordinary” editor of the UMLS should be permitted to assign such a pair of semantic types to a concept. Rather, the approval of a supervisor or the vote of a team of editors should be required for such an assignment. If approval is granted, then this pair will be categorized as “more research required,” until six concepts have been assigned this combination.

Hence, having our adviseEditor system in use by UMLS editors would have warned them concerning the introduction of categorization errors and would have avoided the resource-intensive efforts to correct them. It is especially noteworthy that many of these erroneous combinations of semantic types in Table 6 were reintroduced after already having been corrected and removed once before.

Obviously, an assignment of a pair of semantic types violating any of the other categories of exclusion rules will always be denied. As noted in the Results Section, semantic type assignments that contradict explicit exclusion rules were found in the UMLS. Our comparisons of two versions (2007AC and 2009AA) of the UMLS showed encouraging results, in that many of those erroneous assignments had disappeared. However, in 2011AA new problems were introduced. This shows the urgency of using a system such as adviseEditor for approving new pairs of semantic types.

Some small intersections, categorized by us as “more research required” turned out to be legitimate combinations of semantic types. Over time, their extents have increased and may increase further with the addition of new concepts into the UMLS. When there are six concepts assigned such a combination, it will be categorized as “implicit inclusion.”

Altogether, there are 199 pairs of semantic types that have been assigned to concepts. The sizes of the intersections of their extents vary from 1 to 82,059. The 15 pairs of semantic types with the largest extent intersections and the numbers of concepts in the intersections of their extents are shown in Table 8. These are all intersections with more than 1300 concepts. Each of these intersections involves one semantic type which is a Chemical Viewed Functionally and one semantic type which is a Chemical Viewed Structurally. These largest intersections demonstrate the prominence of the family of inclusion rules defined by Chemical Viewed Structurally and Chemical Viewed Functionally in Section 4.1.1.

Table 8.

Large intersections of extents

Functionally Viewed Chemical Semantic Type	Structurally Viewed Chemical Semantic Type	Size of Intersection Extents
Pharmacologic Substance	Lipid	1475
Pharmacologic Substance	Carbohydrate	2053
Pharmacologic Substance	Inorganic Chemical	2096
Pharmacologic Substance	Nucleic Acid, Nucleoside, or Nucleotide	2351
Hazardous or Poisonous Substance	Organic Chemical	2749
Pharmacologic Substance	Steroid	3110
Antibiotic	Organic Chemical	3414
Receptor	Amino Acid, Peptide, or Protein	4018
Biologically Active Substance	Organic Chemical	4321
Indicator, Reagent, or Diagnostic Aid	Organic Chemical	4684
Pharmacologic Substance	Amino Acid, Peptide, or Protein	6796
Immunologic Factor	Amino Acid, Peptide, or Protein	14064
Enzyme	Amino Acid, Peptide, or Protein	25250
Biologically Active Substance	Amino Acid, Peptide, or Protein	46708
Pharmacologic Substance	Organic Chemical	82059

For future work, a usability study for the adviseEditor system is planned. The Semantic Network is viewed as an “abstraction network” for the Metathesaurus of the UMLS. In recent years"abstraction networks” were derived for several other terminologies, e.g. taxonomies for SNOMED and NCIt [22–24], a schema for the Medical Entity Dictionary (MED) of Columbia [25] and the Specialty Chemical Semantic Network for the Chemical component of the UMLS Metathesaurus [26]. With the introduction of abstraction networks for other terminologies, the need for similar research for such terminologies may arise.

In summary, we note that the adviseEditor system reflects the extensive semantic type knowledge that was implemented in the UMLS over a long period of time by numerous editors. In this way, the adviseEditor system is also serving as a channel for making the valuable experience of generations of UMLS editors available to the current and future UMLS staff members.

6 Conclusions

In the past, there was no systematic account of all combinations of semantic types that are either supposed to be exclusive or supposed to be inclusive. Rather, this information was distributed throughout definitions and usage notes of semantic types. Furthermore, many exclusion rules were not made explicit, as they were assumed to be “obvious” based on some outside source of information, such as the Linnaean taxonomy of animals.

We have collected and organized all such rules into eight rule-categories. We have implemented those rule-categories in the Web-based adviseEditor system that categorizes pairs, triples, quadruples and quintuples of semantic types in batch mode and in interactive mode, and we have computed the numbers of members for each rule-category for pairs of semantic types.

Many interesting cases of the 8778 possible combinations of pairs of semantic types were discussed. Furthermore, we have presented examples of concepts that violate the given exclusion rules. Some of those erroneous semantic type assignments to concepts were introduced only recently. It is hoped that the presented adviseEditor system will be used in the future when extending the UMLS with new concepts, to avoid the introduction of such invalid semantic type assignments.

Highlights.

• Which pairs (triples,…) of UMLS semantic types are permissible, prohibited, etc.?

• We identify kinds of inclusion rules and exclusion rules for pairs (triples, etc.).

• We designed an algorithm adviseEditor based on the eight rule-categories.

• A Web-based system was developed to implement the adviseEditor algorithm.

• The system might help UMLS editors to prevent erroneous semantic type combinations.