Combining rules, background knowledge and change patterns to maintain semantic annotations

Abstract

Knowledge Organization Systems (KOS) play a key role in enriching biomedical information in order to make it machine-understandable and shareable. This is done by annotating medical documents, or more specifically, associating concept labels from KOS with pieces of digital information, e.g., images or texts. However, the dynamic nature of KOS may impact the annotations, thus creating a mismatch between the evolved concept and the associated information. To solve this problem, methods to maintain the quality of the annotations are required. In this paper, we define a framework based on rules, background knowledge and change patterns to drive the annotation adaption process. We evaluate experimentally the proposed approach in realistic cases-studies and demonstrate the overall performance of our approach in different KOS considering the precision, recall, F1-score and AUC value of the system.

Introduction

In order to support various tasks in medical information systems such as retrieving, sharing and exchanging information, the data stored within these systems are usually annotated with concept codes from standard Knowledge Organization Systems¹ (KOS). These annotations make the semantics of the data explicit for machines in order to automatize the above-mentioned tasks. However, KOS evolve over time and their elements may be modified, which, in turn, may affect the dependent annotations as shown in our previous work² and drastically impact the reliability of applications exploiting them. Methods and tools are therefore required to keep semantic annotations updated as KOS evolves. The idea is to avoid a complete re-annotation of the document, which is both time consuming and requires human intervention for validation purpose.

Over the past decade, several approaches have been developed to tackle this problem. For sake of clarity, we split this complex problem into two sub-problems: (1) how to detect the annotations impacted by the evolution of the KOS; and (2) how to update impacted annotations to keep them consistent with the new version of the KOS. Many existing approaches focus on the first sub-problem, proposing several different techniques such as logic rules^3–5 or NLP processes to detect changes^6,7 or even database versioning^8,9. The second sub-problem is mainly addressed by removing the outdated annotations from the corpus⁷, or by adapting existing annotations. By applying the former approach, the user accepts loosing part of the existing knowledge. To avoid this, the latter approach can be adopted. Different adaptation methods are proposed and can be classified according to the techniques used. In⁹, the authors proposed a migration algorithm. Frost et. al.¹⁰ proposed a novel algorithm for optimizing gene set annotations using entropy minimization over variable clusters (EMVC). This filters annotations for each gene set to remove inconsistencies. In their work, Traverso Ribon et al. proposed the AnnEvol¹¹ framework to describe datasets of ontology-based annotated entities. These approaches usually focus on one kind of KOS (e.g. Gene Ontology), or on very expressive ontologies to use reasoning techniques³, and do not exploit information that can be acquired from the analysis of KOS evolution. In our previous work, we designed the DyKOSMap framework for maintaining valid the semantic mappings between evolving KOS¹². However, unlike mappings that formally connect concepts, semantic annotations refer to information of a different nature (e.g. text, images, video …) that is less rich from a semantic point of view. In consequence, the approach that we proposed for mappings needs to be redesigned to address the semantic annotation maintenance problem.

To overcome the limitations of existing approaches mentioned above, our contribution consists of: (i) the definition of a set of rules following a rigorous analysis of the evolution and adaptation of a set of annotations over a 10-year period² (ii) the combination of these rules with two other techniques to improve the quality of the maintenance process: The first technique relies on the use of background knowledge (BK)¹³ while the second exploits semantic change patterns (SCP)¹⁴ that have shown great capabilities for the maintenance of ontology mappings¹². Moreover, we provide an experimental assessment of our framework by maintaining a corpus of documents annotated with ICD-9-CM, MeSH, NCIt and SNOMEDCT.

The remainder of the paper is structured as follows: The following section presents our method for adapting semantic annotations. Then, we introduce the experimental evaluation of our approach and the results we obtained at validation time. Finally, we discuss the results and conclude our paper.

Semantic Annotation Adaptation

In this section we describe our rule-based approach for adapting outdated semantic annotations and the material to evaluate it. We represent annotations following the model of the W3C Web Annotation Data Model¹. We included some modifications, e.g., a property to link the evolved annotation to its past version, the KOS change which impacted the annotation and other properties². To illustrate how an annotation is represented in our system, Table 1 contains the original annotation (from 2009) in the first line and the evolved annotation (from 2010) in the second line.

Table 1:

Example of an evolving annotation, extracted from our silver standard.

KOS	Doc.	Concept	Annotation	Start	End	Prefix	Suffix	KOS label
MeSH 2009AA	232	D019684	Magnoliophyta	4587	4600	during the evolution of	(angiosper ms) [5]. typical	Magnolio phyta
MeSH 2010AA	232	D019684	Magnoliophyta	4587	4600	during the evolution of	(angiosper ms) [5]. typical	Angiospe rm

In our analysis, we used the following features: the name and version of the KOS, the reference of the document used as a resource, the concept code, the annotated text followed by the start and end offset, i.e., the position in the document where this annotation can be found, and the prefix and suffix, i.e., the information that comes before and after the annotation. However, if available, more metadata can be included, respecting our annotation form². The illustrative example shows one annotation produced with the 2009AA MeSH version using the PubMed document 232² and the concept D019684. The annotated text is “Magnoliophyta”, and this can be found in the position [4587,4600]. We set up the system to have four words as a prefix “during the evolution of” and a suffix “(angiosperms) [5]. typical”. We can observe that the concept label used to annotate the text changed from 2009AA to 2010AA.

Our maintenance process relies on the combined use of seven rules, background knowledge and semantic change patterns. The rules proposed derive from our analysis of the evolution of annotations impacted by changes in the underlying terminology². The rules are part of the implementation of our global framework¹⁵. The decision to execute a rule depends on several criteria, but instead of checking all annotations, a first filter is applied: only annotations associated to concepts that have changed (computed with COnto-Diff¹⁶) are checked. Unlike rules for mapping maintenance¹², the proposed annotation maintenance rules consider many other features used to characterize semantic annotation². We based our approach on the guideline associated with semantic annotation¹⁷. The criteria that we used to guide us during the annotation process also inspired us to define the sequence and conditions that our rules have to be applied.

The seven proposed rules are listed in Table 2. Each column represents one feature: the original concept code and the KOS version (CP_v0), annotated text (Annot_v0), a prefix (Prefix_v0), and a suffix (Suffix_v0). We also added one column to show the changes observed (Changed_v1) and another to indicate the rule executed for the presented situation (one rule by line). The proposed rules are:

1. MergeAnnot: This rule will be applied when two parts of a document, annotated with different concepts, can be put together and annotated with one (more specific) new concept only. For instance, in 2004AA, the texts “pregnancy” and “hypertension” were annotated with the concept codes D011247 and D006973, respectively. In 2005AA, text containing both terms, i.e. “pregnancy-induced hypertension” was annotated with the new concept D046110, see Table 2.

2. IncreaseAnnot: This rule increases the amount of information that can be annotated after the evolution of the underlying KOS. To do this, we compare the new label or attribute values of the candidate concept with the information surrounding the initial annotation (i.e., we take into account the prefix and suffix of the annotated text). Concretely, this action modifies the offset value in the annotation model and (if needed) the concept ID, e.g., {D002403, cathepsin} ↔ {D056668, cathepsin l}, see the second example on Table 2.

3. ResurrectAnnot: In some cases, one concept can be temporarily deleted from a KOS, leading to the deletion of the associated annotation. For instance, the annotation chemiluminescence in Table 2 was removed by a change in MeSH 2005AA. This rule allows the annotation to be re-activated when the concept is re-integrated to the KOS, e.g., the concept D017083 in MeSH 2006AA.

4. PluralAnnot: This rule verifies whether the change in the underlying concept or attribute value is due to a plural or singular (agglutination ↔ agglutinations). In this case, the change in the terminology does not imply a change in the impacted annotations since the semantics of the concept are not altered. Note that plurals are language-dependent rules and we are evaluating only for English KOS.

5. ChangeConceptAnnot: This rule changes the concept ID of the annotation due to the evolution of the concept. This situation arises when the label or the attribute value of the concept, used to create the annotation, is moved to another concept or used to create a new concept. For instance, concept D003704 changed to D057174 (referring to Semantic dementia) in MeSH 2009AA/2010AA.

6. SplitAnnot: This rule splits an existing annotation if the evolution of the underlying concept leads to the creation of two more precise annotations. For instance, the text “diabetic foot ulcers”, annotated in 2005AA with the MeSH code D017719, was split into two other annotations in 2006AA: D017719 (diabetic foot) and D016523 (foot ulcers), see Table 2.

7. SuperClassAnnot: This rule changes the concept ID to the superClass ID since no concept can be found to precisely maintain the annotation. It will also change the relation (i.e., “Equivalent” → “Is A”) between the concept and the annotation. For instance, after checking whether any of the previous rules were executed with the annotation infective agents, the last example in Table 2, instead of deleting the annotation, we propose to using superClass to annotate the text. Thus, infective agents is a kind of other organism groupings. Note that it is only possible if the formalism used to annotate the text follows our proposed formalism².

Table 2:

Example of annotations computed by our rules. (CPv₀) concept code in specific year, (Annotv₀) annotated text, (Prefixv₀) prefix, (Suffixv₀) suffix, Changedv₁ result of applied rule.

CPv0	Annotv0	Prefixv0	Suffixv0	Changedv1	Rule
D011247 in 2004AA D006973 in 2004AA	pregnancy hypertension	diabetes mellitus and pregnancy-induced	-induced hy-pertension. Apgars were	{D046110, pregnancy-hypertension}, induced 2005AA	MergeAnnot
D002403 in 2009AA	cathepsin	responses [67]. a	l-like gene (ee049537) has	{D056668, cathepsin l}, 2010AA	IncreaseAnnot
D017083 in 2004AA	chemiluminescence	of western blot	were acquired	{D017083, chemilumines-cence}, 2006AA	ResurrectAnnot
D000371 in 2009AA	agglutination	of antibodies. weakly reactive	required an adequate light	{D000371, agglutinations}, 2010AA	PluralAnnot
D003704 in 2009AA	Semantic dementia	frontotemporal dementia pnfa	?prion	{D057174, semantic dementia}, 2010AA	ChangeConceptAnnot
D017719 in 2005AA	diabetic foot ulcer	associated with	are recom-mended	{D017719, diabetic foot} {D016523, foot ulcers}, 2006AA	SplitAnnot
C50922 in 2009AA	infective agents	the most common	, the necrotic base of	{C14376, other organism groupings}, 2010AA	SuperClassAnnot

The sequence in which the rules are executed is important to assure the quality of the modified annotations. Based on the propositions of the annotation guidelines¹⁷, we established the following sequence: MergeAnnot, IncreaseAnnot, ResurrectAnnot, PluralAnnot, ChangeConceptAnnot, SplitAnnot, SuperClassAnnot. We ranked first the rules that increase the information of an annotation (i.e., MergeAnnot and IncreaseAnnot), as suggested in the guideline “Annotate the most specific concept that correctly describes the disease mention”. The next rules (ResurrectAnnot, PluralAnnot and ChangeConceptAnnot) are mainly related to the structure of the KOS and text. We started by ResurrectAnnot because changing the concept ID (ChangeConceptAnnot) would increase the complexity in identifying the restoration of the concept. The PluralAnnot rule is an exception, because it does not affect the other rules and can be placed anywhere in the sequence. The SplitAnnot was placed close to the end of our process due to the rare cases where it occurs as mentioned by Doğan et. al.¹⁷. It respects the following recommendation: “Annotate a disease mention using multiple concepts to logically describe the disease mention, using the “+” concatenator”. Finally, the SuperClassAnnot was positioned at the end of our process as an alternative to the removal of the annotation.

The precision of rules has some limitations. We observed that some exceptions could sensibly increase the complexity of the rules (and the time required to execute them). Thus, we decided to evaluate other potentially complementary methods in order to improve the quality of our outcomes. In this sense, we selected two other methods: background knowledge and semantic change patterns. Details of each method can be found in¹³ and¹⁵, respectively. We briefly introduce the main aspects of these two methods below:

Background Knowledge (BK)

The main idea behind BK is to use information inferred from external ontologies in order to discover the semantic relation between two successive versions of a concept¹³. For instance, the evolution of the label “Magnoliophyta” to “Angiosperms” cannot be characterized only by considering the syntactic aspect. However, external “sources of knowledge” can tell us that these two terms are synonyms. We used the mappings between the concepts from different terminologies to determine the relation between these concepts. In our case, we used the mappings contained in Bioportal¹⁸. However, this approach can be extended to any external resource.

The BK algorithm (see algorithm 1) presents an overview of the whole process. Fig. 1 helps to illustrate how the algorithm works. The input of the algorithm is the concept ID (C_s), label (L_s), KOS target (KOS_t), and KOS source (KOS_s), e.g., (D019684, Magnoliophyta, MESH, {SNOMEDCT, ICD-9-CM, NCIT}). After initializing the variables (lines 1 to 2), our method queries external sources using the impacted annotation (line 3) label and stores the resulting concepts (Request). For instance, the concept 420928000, from SNOMEDCT, is one candidate. Only concepts candidates belonging to the source KOS KOS_s are kept (lines 4-5). Then, for each concept from Request, the mappings are collected (line 6). Only mappings to the target KOS (KOS_t) are kept (lines 7-10), they are the gray boxes in Fig. 1. From all candidates that satisfy the previous conditions, only the best candidate is selected to maintain our annotation (lines 11-13). The selection criteria is based on the semantic similarity (i.e., Tversky similarity) between the concept source and the concept target, i.e., (C_s and C_t) from MESH.

Figure 1.

Use of BioPortal as Background Knowledge

Algorithm 1: Background Knowledge to find the link between 2 concepts

Input: Concept source Cs; Label Ls; Ontology source KOSs; Ontology Target KOSt

Output: Concept Target Ct

1 MappingSet ← Ø

2 Result ← Ø

3 Request ← getConceptsFromBK(Ls)

4 forall cp ∊ Request do

5 if (cp ∊ KOSs) == TRUE then

6 MappingSet ← getMappings(cp)

7 forall mapping ∊ MappingSet do

8 target ← getConceptTarget(mapping)

9 if (target 2 KOSt) == TRUE then

10 Result ← target

11 forall obj ∊ Result do

12 calSemanticDistance(Cs; obj)

13 Ct ← getHighSimilarity(Result)

14 return Ct

Semantic Change Pattern

Change Patterns are morphosyntactic modifications observed in attribute values of a concept, using linguistic-based features to identify the correlation between concepts over time. This technique has already been explored in the context of ontology mapping adaptation¹⁴ and we adapted it to the maintenance of annotations¹⁵. It consists of a set of rules that allow the identification of the impact of evolving an ontology on the semantics of the concept.

For instance, the annotation, “Physiologic processes” produced using MeSH 2008AA, was impacted in 2009AA. This is due to a change in the attribute value in the definition of concept D010829 leading to “Physiological Phenomena”. The rules proposed in¹⁴ will determine which class of change it belongs to: total copy, total transfer, partial copy, partial transfer. We can conclude, for instance, that if changes in the ontology lead to a total transfer of information from one concept to a new one, than the annotation will probably change the source (or target) concept to this new one too. This is a simplification of inferences that can be used. To better understand the context and the complexity behind SCP, we advise reading¹⁴.

Experimental Assessment

In this section, we introduce the method and material we have used to evaluate our approach. The experiments we have conducted consist of applying our approach to a set of annotated documents and comparing it to a corpus of reference representing an evolved version of the initial set of documents.

Material

Our annotation maintenance process takes as its input:

• a set of outdated annotations,

• the old and new OWL versions of the KOS used to generate the annotation.

In our experiments we have used: International Classification of Diseases, Ninth Revision, Clinical Modification (ICD- 9-CM), Medical Subject Headings (MeSH), National Cancer Institute Thesaurus (NCIt) and Systematized Nomenclature of Medicine - Clinical Terms (SNOMEDCT). We used versions 2009AA and 2010AA downloaded from the UMLS and transformed into OWL files. These KOS follow the formality expressed in the UMLS metathesaurus³, e.g., (Concept A rdfs:superClassOf Concept B); (Concept A skos:prefLabel term 1), etc. We used COnto-Diff¹⁶ to identify the changes in the new version of the KOS since these changes are strongly correlated with the validity of annotations².

Silver Standard

Since no annotation baseline generated with sequential ontologies versions exists, we had to build our own corpus of reference using the annotations produced in² as a basic resource. To do this, we randomly selected 500 annotations generated with the 2009AA version of the four KOS (125 annotations from each KOS) and asked three experts to manually validate/correct the evolution of the 500 selected annotations, according to the 2010AA version of the corresponding KOS. Each expert validated 1/3 of the annotations and no discussions between them was organized. The consolidated outcomes compose the silver standard, which can be downloaded from http://www.elisa-project.lu/, look into menu publications/downloads. We adopt the term “silver” to indicate that our reference is based on only one viewpoint.

To measure the effectiveness of the proposed approach, we used classic well-known metrics from the literature, such as, precision, recall, F1-score, ROC curve, accuracy, false/true positives/negatives. For the sake of readability, we will present only three metrics in the table. But, we used all six metrics to investigate/understand two characteristics of our method: i) the capacity of our framework to detect impacted annotations after changing a KOS concept; and ii) the ability to correctly evolve those impacted annotations into consistent ones. In this case, consistency means equivalency with the silver standard. We measured the efficiency of the rules alone, the BK alone, and the SCP alone and the efficiency of the combined techniques in order to determine whether they complement each other.

Results

When applying the three annotation maintenance methods (BK, SCP, Rules) to our dataset we can observe a significant difference in the results (based on the six criteria used). For instance, as shown in Table 3, all three methods can provide good precision, but there is a significant variation [0, 0.98] regarding the recall. In the first line of Table 3, we present the precision, recall, and F1-Score resulting from applying the BK method to four different subsets of our initial dataset (ICD-9-CM, MeSH, NCIt, and SNOMEDCT). We also evaluate the consequence of combining the methods (2-by-2, and all together). For instance, the fourth line of the table presents the results of combining BK and Rules methods, while the seventh line shows the results of combining all three methods.

Table 3:

Precision (P), Recall (R) and F1-Score (F1) of impacted annotations computed using three different methods (BK, SCP, Rules) and the combination of them. The red and orange colors indicate low and medium recall, respectively.

Method	ICD9CM				MeSH				NCIT				SNOMEDCT
	P	R	F1		P	R	F1		P	R	F1		P	R	F1
BK	1	0.16	0.28		1	0.05	0.51		1	0.13	0.24		0.96	0.54	0.69
Rules	1	0,98	0.99		0.98	0.96	0.95		0.98	0.88	0.93		1	0.79	0.88
SCP	1	0.08	0.15		1	0.02	0.50		0.83	0.10	0.17		0	0	0
BK & Rules	1	0.98	0.99		0.98	0.96	0.95		0.98	0.88	0.93		1	0.79	0.88
BK & SCP	1	0.19	0.32		1	0.05	0.51		0.91	0.19	0.32		0.96	0.54	0.69
Rules & SCP	1	0.98	0.99		0.98	0.96	0.95		0.98	0.88	0.93		1	0.79	0.88
CombineAll	1	0.98	0.99		0.98	0.96	0.95		0.98	0.88	0.93		1	0.79	0.88

The goal of this first set of experiments was to evaluate whether the methods (or a combination of them) provides satisfactory quality (in terms of F1-Score) to determine whether an impacted annotation will evolve or not. Note that we are not yet evaluating whether the annotation evolved correctly (this is part of the second evaluation step). A quick analysis shows that all methods can accurately identify some of the evolving annotation, but not all. From a practical point of view, whether an error of 2% is acceptable for the domain, then KOS engineers can trust our best method to identify annotations that will change (i.e., the minimal observed precision was 98%). However, the recall can be significantly different according to the dataset and the method adopted. We detail the reasons for this in next section. We would like to highlight that our Rules method had an impressive performance, obtaining in some cases a F1-Score of 99%.

The second evaluation process consists of applying methods to select which adaptation actions can make the annotation evolve correctly, and compare the outcomes with the silver standard. The goal is to measure how precise our recommendations are. Table 4 describes the performance of each method regarding the four different datasets. Each experiment is represented by an Area Under the Curve (AUC) value, giving the probability that a randomly selected instance will correctly be adapted by our method¹⁹.

Table 4:

AUC values of developed heuristics used to maintain annotations. The red and blue color highlight the lower and higher values for each dataset, respectively.

Method	ICD9CM		MeSH		NCIT		SNOMEDCT
	AUC		AUC		AUC		AUC
BK	0.613		0.554		0.663		0.708
Rules	0.899		0.850		0.721		0.833
SCP	0.593		0.550		0.606		0.500
BK & Rules	0.915		0.863		0.721		0.833
BK & SCP	0.601		0.554		0.663		0.708
Rules & SCP	0.895		0.838		0.731		0.833
CombineAll	0.923		0.863		0.731		0.833

The AUC values of the analyzed methods vary according to the dataset. A quick analysis in Table 4 shows that combining all methods provides slightly better results than applying only one of them. Furthermore, SCP shows the lowest AUC values of all heuristics. We also verified that the AUC has significant differences between the KOS, like those between ICD-9-CM and NCIt. Detailed explanations on these observations are provided in the next section.

Discussions

The analysis of annotation evolution in the healthcare domain is an understudied topic. As explained in the introduction section, several works propose the automatic detection of inconsistent annotations, but few of them address the automatic correction of inconsistent annotations. The work presented in this paper shows that some annotations can be preserved/adapted after the evolution of the KOS used to generate the annotations. Three methods were proposed: Background Knowledge, Semantic Change Patterns, and Domain Specific Rules. The outcomes presented in the results section demonstrate that we can obtain high AUC by applying these methods together in the automatic maintenance of annotations or to support domain experts in this activity.

When analyzing each method, we observed that BK contributes to the precision of the annotation changes. The main characteristic of BK is that it depends on the richness of information in other sources (e.g., ontologies with overlapping concepts). Furthermore, BK can provide unaligned mappings to past KOS versions leading to another phase to filter inconsistent results. This limitation of the method generates a low recall (but good precision) of results. We estimate that it can be an interesting complementary method for the maintenance process. Another aspect that can be deduced from the experiments is the dependency of the BK method on the expressivity and consistency of the KOS. For example, MESH D002544 has as a synonym the concept labels that are considered siblings in other KOS (e.g. “Cerebral infarct left hemisphere” SNOMEDCT 362323007 and “Cerebral infarct right hemisphere” SNOMEDCT 362322002), leading to loose information when the system follows the KOS mappings that cross MeSH. We also observed that SNOMEDCT and ICD-9-CM allow equal “labels of concepts” (e.g. in SNOMEDCT the concept “diverticulosis” has the following codes 31113003, 397881000, and 68047000), which can lead the system to select the wrong concept to replace the impacted annotations, and necessitating an additional disambiguation phase. Another limitation of BK is that we can only find the last version of KOS in the Bioportal (i.e., from 2016), but our experiments use documents annotated with the version 2009AA and 2010AA. Versionning is an aspect that is not yet integrated to Bioportal, but it deserves to be considered in future.

The analysis of SCP also shows a good precision and low recall. The reason here is that SCP considers only change between concepts that are in the same neighbourhood (i.e., siblings, super-, and sub-concepts). Thus, changes that move the concept to other branches of the KOS are not included, leading to an increased number of false negatives. For SNOMEDCT, Table 4, we did not observe any cases with SCP in our dataset. Since, the data used was randomly selected, we consider that it was a coincidence. We did our analysis based on the results coming from the other KOS². However, this heuristic is able to cover cases where the Rules or BK do not work. For instance, the annotation “ubiquitin carboxy-terminal hydrolase” NCIt C21490 correctly evolved to “ubiquitin carboxyl-terminal hydrolase BAP1”. Thus, the AUC value to NCIt in Table 4 is higher in Rules & SCP than that compared to BK and its combinations (lines 1,4, and 5).

Domain-specific rules are defined to describe frequent patterns of changes, and it is expected to generate outcomes from them with good precision and recall. This was the case for the rules proposed in our experiments. However, our rules do not cover all annotation evolution cases perfectly; some ambiguities are still observed. The proposed rules are not very precise when the annotated text and the concept label (or synonyms) do not have an exact match. For instance, the text “dysarthria” was annotated in 2009AA with the concept 784.5 (Other speech disturbance); in 2010AA, a new sub-concept 784.51 (“dysarthria”) was created. The rules were unable to determine that the specialized concept was more appropriated to the annotation. This evolution was correctly proposed by the BK method. Such cases of misclassification are more frequent in NCIt because we observed fewer annotations with an exact match and a lower number of mappings from/to this KOS in Bioportal.

Another aspect to highlight, is that depending on the context in which the maintenance methods were used (e.g., high expressive KOS), there are considerable differences in the sets of results. We also observed that a combination of methods can be used for a more complete set of evolution situations, as in following:

• SCP and BK methods show low complementary results to identify whether the KOS evolution impacts the annotations, but an improvement was observed by combining the methods to identify the correct evolution of the annotation.

• On one hand, Rules increase the amount of corrected annotations of all BK and SCP analyzed cases. On the other hand, BK and SCP have a minimum effect on the rules for the first set of experiments and demonstrate the ability to improve the identification of correct evolution of the annotation (second set of experiments).

• Compared to the pairwise combination of the methods (BK & Rules, SCP & Rules), applying the three methods together improves (or at least keeps the same) the AUC of the correct evolution of the impacted annotations (second set of experiments).

Conclusion

The work presented in this paper shows the possibility of having annotation maintenance tools that can keep tracking KOS evolution and measure the impact that it will have on the set of annotated documents. Moreover, it also demonstrates that automatic correction/adaptation of annotations can reach a reasonable reliability rate. But, it is important to highlight that the role of human beings is still determinant to assure the quality of the annotations in critical scenarios, as observed in the biomedical domain. Our approach contributes to a more generic objective that intends to define methods and formalism to improve the annotation task in order to better support the annotation maintenance⁴. In previous work², we evaluated the annotation formalism and proposed some extensions to close the identified gaps. In future work, we will evaluate the performance of our approach in a larger temporal window and search for methods that can generate additional rules from the data.