STAT: A Web-based Semantic Text Annotation Tool to Assist Building Mental Health Knowledge Base

Abstract

Mental health problems are serious among American adults and many of them are turning to the Internet for help. However, online mental health information is not well-organized and in low quality. We are building a mental health knowledge base (MHKB) with evidence-based information extracted from scientific literature manually, but lacking efficiency. We envision to leverage collective wisdoms through crowdsourcing to speed up the curation of MHKB. In order to integrate with crowdsourcing platforms, we designed and prototyped a web-based annotation tool, STAT (Semantic Text Annotation Tool), with real-time annotation recommendation and annotation quality analysis, to facilitate management of laypeople annotators recruited through crowdsourcing to complete the necessary annotation tasks.

I. INTRODUCTION

One in 5 adults have a mental health condition in the United States (US); however, 56% of them do not receive or seek medical treatments. One reason is that there is a serious mental health workforce shortage [1]. Further, people often want to keep their mental health issues private and are reluctant to seek help. With the increasing coverage of the Internet and smartphones, the way how people learn about and manage their mental health is changing. People frequently use the Internet to look for mental health information. Nevertheless, existing online information about mental health is poorly organized, not evidenced-based, of poor quality, and confusing to health information consumers. A formal knowledge representation such as a Semantic Web knowledge base (KB, or knowledge graph) can help better organize and deliver quality mental health information and thus fill the gap. In our current work, we are trying to build a mental health KB using high quality, evidence-based resources, such as publications from high impact journals. However, the manual annotation and extraction of semantic triples (i.e., facts) in the form of subject-predicate-object expressions (e.g., “dementia”—is_a—“mental disorder”) from publications’ abstracts are time-consuming and difficult.

In our previous work [2], we demonstrated that crowdsourcing is indeed a low-cost mechanism to collect labeled data from non-expert laypeople. Even though individual layperson might not offer reliable answers, the collective wisdom of the crowd is comparable to expert opinions. To speed up the construction of the mental health KB, we decided to explore crowdsourcing. However, the annotation tasks are laborious and hard to execute, even for experienced annotators, without a good annotation tool.

Thus, we first conducted a review of existing text annotation tools. As shown in Table. 1, these existing tools have various limitations and not optimized for crowdsourcing use: 1) many of these tools are not web-based, thus are not suitable for crowdsourcing use as users need to download the applications first before carrying out the annotation tasks and making coordination of the tasks difficult; 2) some tools do not provide any annotation support such as recommendations (e.g., suggesting the candidate semantic classes of an entity); 3) many tools are outdated and not well-maintained; and 4) some tools do not provide a mechanism for monitoring the annotation quality (e.g., reporting inter-rater agreements) making it inconvenient for crowdsourcing tasks. Although WebAnno 3 [3] meets our basic text annotation requirements, it is not tailored to our use case—annotation of semantic triples for curating semantic web KB. We evaluated its source code and found it is difficulty to modify it to fit our annotation task workflow in a crowdsourcing environment.

TABLE I.

COMPARISON OF EXISTING TEXT ANNOTATION TOOLS

Tool	Web-based	Annotation Recommen dation	Quality Analysis	Last Updated
WordFreak	×	√	×	2013
GATE	√	√	×	2018
Knowtator	×	×	√	2013
Stanford	×	×	×	2018
Atomic	×	×	×	2018
WebAnno 3	√	√	√	2019
Anafora	√	×	×	2018
BRAT	√	√	×	2018
YEDDA	×	√	√	2019
STAT	√	√	√	-

We thus prototyped a web-based, user-friendly Semantic Text Annotation Tool (STAT) purposely built for crowdsourcing use. As a use case for building a mental health KB, STAT interfaces with existing mental health related ontologies from BioPortal [4] and controlled vocabularies from the Unified Medical Language System (UMLS) [5]. Based on these terminology resources, STAT provides real-time annotation recommendations to help both experts and laypeople extract semantic triples from scientific literature. The system also has an administrator interface that provide support for analyzing, monitoring and managing the crowdsourcing annotation quality and results.

II. METHODS

A. Needs Assessment and Function Design

To collect user needs and requirements of STAT, we first interviewed with annotators who have extensive experience on the semantic triple annotation task. We asked the annotators to summarize their annotation workflows, as well as the problems or difficult parts they have encountered during their annotation processes. Most of these annotators did not use any tools in their past annotation tasks. We analyzed their annotation workflow and designed STAT with a list of initial functions. We also proposed a number of convenient functions to address specific pain points raised by the annotators during their interviews. We sketched these functions considering to the feedback collected in these interviews to produce a low-fidelity prototype. Finally, we presented the low-fidelity prototype to the annotators and discussed with them to further improve the design before producing a working prototype of STAT.

B. Implementation of a STAT Prototype

We implemented a working prototype of STAT followed the design sketch generated in the function design phase. As shown in Fig. 1, the system architecture of STAT consists of three parts.

Fig. 1.

System architecture of STAT.

• 1)A database module:The database module consists of two databases. A relational database, PostgreSQL, is used to store and organize user account information as well as the annotation history data which could serve as a good recommendation resource. A NoSQL database, mongoDB, is used to store annotation data in a rich way while being capable of holding different types of metadata freely. The combination of these two kinds of databases improves the scalability and response speed of the system.
• 2)A Python-Flask-based backend:The Python-Flask-based backend provides Representational State Transfer-ful (RESTful) application programing interfaces (APIs) that connect the web-based frontend with the backend databases and the two vocalbulary services through their repsective APIs.
• 3)A web-based frontend:The web frontend is built with the popular Angular JavaScript framework, which consists of an annotation user interface and an administrator user interface.

C. Evaluation

We compared the efficiency of using STAT to annotate semantic triples with our previous workflow (i.e., spreadsheet-based). We identified 3 abstracts related to both “mental health” and “cancer” with different complexity levels, ranging from 9 to 15 semantic triples. Two annotators were invited to annotate these articles. We tracked the annotation time for each abstract and collected usability feedback from the annotators. We asked the annotators to thoroughly read these abstracts before they started the annotation tasks, so that their annotation time was not influenced by annotators’ familiarity with the content. We also switched the orders of the annotation tasks (i.e., spreadsheet- based first and then STAT-based or STAT-based and then spreadsheet-based).

III. RESULTS

A. Results of the Needs Assessment and Function Design

Based on interviews with annotators, we summarized their existing annotation workflow consisting of five steps: 1) collect articles based on keywords (i.e., “mental health” and “cancer” in this case study) from PubMed; 2) sort the publications by impact factor to prioritize tasks; 3) screen each abstract to determine whether it is relevant; 4) annotate entities and relations that form semantic triples; and 5) normalize the entities and relations to the appropriate ontology classes (i.e., searching through BioPortal or UMLS). The last two steps are the most time-consuming and difficult steps.

We presented a number of different design sketches to the annotators based on the initial functional requirements. Fig. 3 shows one design sketch for the annotation interface with a selected text identified as an entity. A list of recommendations will appear in the annotation section based on the selected text.

Fig. 3.

An example of design sketch of STAT.

B. STAT Annotation Module

The annotation module consists of three parts: 1) a text display area, 2) an annotation popup window, 3) and a triple construction panel.

• 1)A text display area:As shown in Fig. 4, the left part is the text display area that shows the text to be annotated. Annotators can read the text and annotate words or phrases as entities (e.g., texts highlighed in blue) or relations (e.g., texts highlighted in yellow). To assign a label to a text, an annotator selects and highlights a block of text first, right-clicks the selected text, and chooses between “entity” or “relation” from the context menu to indicate its an entity or relation. The users can also use keyboard shortcuts, “Ctrl + E” for entity or “Ctrl + R” for relation, to assign the label. The annotator can remove an annotation by clicking the delete button on the right side of an annotated text.
• 2)An annotation detail window:As shown in Fig. 5, when the annotator assigns a label to a text or click an existing annotation, an annotation window will pop up, where the annotator can normalize the text to a standardized term. The annotation window also allows the annotator to look up for the definition of the selected text using “Google” or “Merriam-Webster Dictionary”. As shown in Fig. 6, when the annotator clicks the annotation dropdown, the system will provide a list of recommended terms based on three sources: 1) the annotator’s annotation history, 2) ontologies from the BioPortal, and 3) terms from the UMLS vocabulary databases.
• 3)A triple construction panel:As shown in Fig. 4, the right part of the interface is the semantic triple construction panel. The annotator can drag an annotated text from the text display area and drop it into one of the three semantic triple boxes (i.e., subjet-predicate-object). Fig. 7 shows an example where all the three triple boxes are filled and ready to be confirmed as a semantic triple. When a triple is confirmed, it will be added into the triple table. The annotator could delete a triple by click the “Delete” link associated with each triple.

Fig. 4.

The main annotation interface: 1) a text display area on the left, and 2) a triple construction panel on the right.

Fig. 5.

An example of the annotation detail window for a selected text “depression”.

Fig. 6.

Annotation recommendations for the text “depression” based on UMLS terms.

Fig. 7.

An example of adding a complete triple.

C. STAT Administrator Module

The STAT administrator module consists of three functions: 1) annotation task management (e.g., creation and assignment); 2) annotation progress monitoring; and 3) analysis of annotator agreements for quality assessments.

We used a heatmap to visualize inter-rater agreements. Fig. 8 shows an example of the inter-annotator agreements among 5 annotators on the entity level (i.e., agreements can be calculated on entity-, relation-, or semantic triple-level). It is clear that annotations from “Annotator 1” are different from the other five annotators indicating quality issues with Annotator 1.

Fig. 8.

An example of an entity-level inter-annotator agreement heatmap.

D. Evaluation Results

The total time of annotating all 3 abstracts is 33.8 minutes (i.e., average between two annotators) using previous workflow and 24.3 minutes using STAT (i.e., a 28.1% time reduction). For the abstract with the most number triples, using STAT leads to a 35.3% time reduction (from 20.1 minutes to 13.0 minutes). We noticed that STAT has a bigger advantage as the number of entities, relations, and triples in an abstract increase. This is possibly due to the recommendation system that automated the normalization task through interfacing with BioPortal and UMLS. Both annotators reported that 1) they liked the recommendation system, especially the recommendations based on their own annotation history; 2) the interfaces with BioPortal and UMLS saved their valuable time; 3) but, we need to set a limitation to the number of recommended terms, as it is difficult to locate the best term with too many recommendations; and 4) they wish to pre-select the target ontologies and/or UMLS source vocabularies on a project by project basis.

IV. DISCUSSION AND CONCLUSION

We prototyped a web-based semantic text annotation tool, STAT, tailored to extract semantic triples from scientific literature for curating Semantic Web knowledge bases. Even though our initial work focused on creating a mental health knowledge base, STAT can be readily used in other disease domains. Such as tool is much needed because of the rising needs to curate evidence-based knowledge bases for organizing online health information. In our future work, we will follow an iterative user-centered design approach to further tailor STAT to meet end-user needs and test its usability in a crowdsourcing environment. One potential improvement is to incorporate a machine learning-based information extraction component that can recommend candidate entities to be extracted. Such a process turns the task of “extraction” into “verification”, which can significantly improve extraction efficiency in a crowdsourcing environment.