Development and Evaluation of an Ontology for Guiding Appropriate Antibiotic Prescribing

Abstract

Objectives

To develop and apply formal ontology creation methods to the domain of antimicrobial prescribing and to formally evaluate the resulting ontology through intrinsic and extrinsic evaluation studies.

Methods

We extended existing ontology development methods to create the ontology and implemented the ontology using Protégé-OWL. Correctness of the ontology was assessed using a set of ontology design principles and domain expert review via the laddering technique. We created three artifacts to support the extrinsic evaluation (set of prescribing rules, alerts and an ontology-driven alert module, and a patient database) and evaluated the usefulness of the ontology for performing knowledge management tasks to maintain the ontology and for generating alerts to guide antibiotic prescribing.

Results

The ontology includes 199 classes, 10 properties, and 1,636 description logic restrictions. Twenty-three Semantic Web Rule Language rules were written to generate three prescribing alerts: 1) antibiotic-microorganism mismatch alert; 2) medication-allergy alert; and 3) non-recommended empiric antibiotic therapy alert. The evaluation studies confirmed the correctness of the ontology, usefulness of the ontology for representing and maintaining antimicrobial treatment knowledge rules, and usefulness of the ontology for generating alerts to provide feedback to clinicians during antibiotic prescribing.

Conclusions

This study contributes to the understanding of ontology development and evaluation methods and addresses one knowledge gap related to using ontologies as a clinical decision support system component—a need for formal ontology evaluation methods to measure their quality from the perspective of their intrinsic characteristics and their usefulness for specific tasks.

1. Introduction and Background

An ontology is a knowledge structure used to formally represent and share domain knowledge through the modeling and creation of a framework of relevant concepts and the semantic relationships between those concepts [1–3]. Ontologies explicitly structure and represent domain knowledge in a machine-readable format that humans are capable of understanding; thus, they can be incorporated into computer-based applications and systems to facilitate data annotation [4–6], decision support [6, 7], information retrieval, and natural-language processing [4, 8] and serve as an integral part of the Semantic Web [5, 9]. Ontologies also have the potential to support the development of clinical decision support (CDS) systems in a manner that enhances reusability of data and knowledge.

Developing ontologies according to ontological development principles can potentially facilitate interoperability and reusability. Conversely, designing ad-hoc ontologies without the use of development standards has created an environment where numerous ontologies exist and there is limited capacity to communicate and reuse the knowledge represented in them across disciplines [1, 10, 11]. Although there is no consensus on how to develop ontologies, several approaches have been described with most sharing some common development elements. Likewise, while formal evaluation methods have the potential to maximize the benefits of ontologies, within the computer science, philosophy, and biomedical ontology communities there is no standard approach to evaluating the quality of ontologies from the perspective of their intrinsic characteristics or extrinsic value (i.e., usefulness for a specific task) [6, 12–17].

Antimicrobial resistance is a global health problem that is exacerbated by the misuse of antibiotics [18–21]. Incorporating a CDS system into antimicrobial stewardship prescribing programs is an effective, although underutilized, approach to address inappropriate antibiotic prescribing [22–26]. Antibiotic CDS systems are often locally developed, which can be expensive, time-consuming, and difficult with respect to sharing and reusing knowledge bases. Implementing an ontology as a component of an antibiotic CDS system can facilitate the creation and sharing of application-independent CDS modules that can be implemented into localized environments; however, development of ontology-based antibiotic CDS systems to support therapeutic planning has not been fully explored.

The aims of this study were to develop and apply formal ontology creation methods to the domain of antimicrobial prescribing and to formally evaluate the resulting ontology through intrinsic and extrinsic evaluation studies.

2. Methods

Study methods include those for ontology development, intrinsic evaluation of the ontology for correctness, extrinsic evaluation of the ontology for usefulness, and the development of three artifacts (prescribing rules, alerts, an ontology-driven alert module, and a patient database). The results of the application of each of these methods are presented in the Results section. The Columbia University Institutional Review Board approved all research procedures. All user evaluations were conducted in a laboratory setting.

2.1. Ontology development

We used two guides [2, 27] to define a six-step development process for the antimicrobial-microorganism ontology: define the ontology domain and scope; review existing ontologies; identify classes and properties; create a conceptual map; identify and implement an upper ontology; and implement the ontology in a formal representation.

2.1.1. Define the ontology domain and scope

The ontology domain and scope were defined using requirement assessment techniques. Two focus group sessions with 24 infectious diseases (ID) and non-ID clinicians (Resident, Fellow, and Attending Physicians, Nurses, Nurse Practitioners, and Pharmacists) yielded 13 functional requirements of a computerized antibiotic therapeutic planning tool. The clinical scope of the ontology was guided by three semi-structured interviews with an ID Attending Physician, an ID Pharmacist, and pharmacy department data pertaining to antibiotic usage and ID questions frequently called into the antibiotic approval pager.

2.1.2. Review existing ontologies

Prior to constructing the ontology, the biomedical literature and ontology repositories such as the NCBO BioPortal [17], Protégé Wiki (http://protegewiki.stanford.edu), and Ontology Biomedical Ontologies (OBO) Foundry [10] were searched to identify potential candidate ontologies for the chosen domain. A review of 20 existing ontologies for the content coverage and depth of assertive knowledge for antimicrobial agents, microorganisms, and infectious diseases was conducted. While the Generalized Architecture for Languages, Encyclopedias, and Nomenclatures in Medicine (GALEN) [28], Medical Entities Dictionary (MED) [29], and Systematized Nomenclature of Medicine-Clinical Terms (SNOMED CT) [30] included classes representing antimicrobial agents, infectious diseases, microorganisms, and assertive knowledge; the Sanford Guide to Antimicrobial Therapy [31] was used as the primary source for content as we determined that it provided the best comprehensive coverage of the domains of interest and the rich assertive knowledge needed for the scope of this ontology.

2.1.3. Create classes and properties

We followed the middle-out approach [2, 3] in which general concepts from the initial list of classes and properties were derived from the functional requirements to produce high-level classes for an upper hierarchy and sub-classes representing more specific details. Ontology content was supplemented by the NewYork-Presbyterian Hospital (NYP) empiric antibiotic therapy guidelines for community-acquired pneumonia (CAP) and urinary tract infection (UTI) and NYP ID guidelines for the directed treatment of organisms from blood, respiratory, and urine cultures to provide local context. The RacerPro reasoner [32] was utilized to provide automated reasoning for consistency checking and inferred classification of the ontology classes.

2.1.4. Create a conceptual model

A hierarchical conceptual model was created to aid in the organization and design of the ontology classes and properties prior to implementing the ontology in a formal representation. The conceptual model was designed with the Institute for Human and Machine Cognition CmapTools application [33] and revised throughout the development process to ensure clarity, coherence, extendibility, reduce redundancy [3, 34] and to support the functional specifications.

2.1.5. Select and implement and upper ontology

To maximize the interoperability and reusability of our ontology with existing and future ontologies, the Basic Formal Ontology (BFO) [35] was implemented as the upper ontology. BFO promotes interoperability across a variety of biomedical domain ontologies for scientific research, provides a basic structured framework generalizable to multiple application domains without the inclusion of domain-specific concepts, and has been extensively tested within the biomedical domain. BFO was also available in a formal representation that could be easily integrated into our ontology, as opposed to some other biomedical upper ontologies. We used BFO to assist in designing the structure of our ontology and defining the ontology classes and properties.

2.1.6. Implement the ontology in a formal representation

The ontology was represented in the Web Ontology Language (OWL) [36] by one investigator (TJB) using the Protégé-OWL editor [37], which is an extension of the Protégé platform [38] and enables the creation and representation of ontologies in OWL in one step. Protégé was selected because it is an actively maintained open-source application, supports automated reasoning tasks such as consistency checking and automatic classification of classes using description logic (DL) expressions [39], and facilitates reasoning about individuals in the ontology using the Semantic Web Rule Language (SWRL) [40]. Moreover, this choice enabled the development of customizable plug-ins to extend Protégé, and Protégé has a large community of users who contribute to the development of Protégé-OWL applications. An expert Protégé user provided regular feedback on the ontology development.

2.2. Evaluation of ontology correctness (Intrinsic evaluation)

A set of ontology design principles and domain expert review were applied to measure ontology correctness in terms of structure and content.

2.2.1. Ontology design principles

Cimino’s Desiderata [13] were used as the set of desired design qualities for the ontology. Although the Desiderata were originally delineated in the context of controlled medical terminologies, the qualities are also of relevance to ontologies: content, concept orientation, concept permanence, nonsemantic concept identifiers, polyhierarchy, formal definitions, reject “not elsewhere classified,” multiple granularities, multiple consistent views, context representation, graceful evolution, and recognize redundancy. Three qualities (content, concept permanence, and graceful evolution) were not considered in the intrinsic evaluation given that the ontology was newly created. The evaluation was conducted by comparing the ontology classes with the desired quality to determine if each was satisfied in the ontology.

2.2.2. Domain expert review

Domain experts assessed the correctness of the ontology in representing domain concepts (e.g., antibiotic agents, microorganisms, and infectious diseases) and the relationships among them via the “laddering” technique [41, 42]. Two ID experts (an ID Fellow and ID Pharmacist) were recruited via email and received financial compensation for their time. Two structured laddering guides were created based upon the local ID guidelines for CAP (16 questions) and UTI (12 questions). An example of a question for a downwards probe for the isTreatedWithPrimaryTherapy relationship is, “Could you tell me some types of antibiotic agents suggested as primary empiric therapy for catheter-associated UTI?” The laddering guide for catheter-associated UTI is displayed in Appendix A.

During the evaluation session, participants reviewed a subset of the ontology that focused on the empiric treatment of CAP and UTI. The structured questions served as probes to elicit the participant’s knowledge about CAP and UTI as input to the hierarchy. As the participant answered a question, the facilitator constructed the hierarchy graphically using Post-it® notes to represent the ontology classes for causal pathogens, antibiotic treatments, and antibiotic mechanisms of action. This approach provided visual representation of the hierarchy as it was created and immediate feedback to the participant of where he/she was in the hierarchy. Each session lasted for approximately 30 minutes and was audio-recorded using a tape recorder. Photographs of the hierarchy were taken after the laddering session and the hierarchies created by the ID experts were compared with the antimicrobial-microorganism ontology.

2.3. Evaluation of ontology usefulness (Extrinsic evaluation)

The usefulness of the ontology was assessed for the tasks of knowledge management and generation of antibiotic prescribing alerts. This required the development of three artifacts: a set of prescribing rules, alerts and an ontology-driven alert module, and a patient database. The artifacts were thoroughly designed in order to conduct the extrinsic evaluations. Subsequently, the two usefulness studies were conducted.

2.3.1. Development of Artifacts

2.3.1.1. Rule development

The prescribing rules were represented in SWRL, a former World Wide Web Consortium (W3C) proposed standard rule language for the Semantic Web that is based on OWL [43]. SWRL rules were written using the Protégé SWRLTab, which is tightly integrated with OWL, and facilitates the creation of rules that can be expressed using OWL classes from the ontology to enhance reasoning capabilities [40, 44]. The SWRLTab enabled the creation of SWRL rules with built-ins for mathematical and string operations, incorporated the Jess Rules Engine (http://www.jessrules.com/) to execute the SWRL rules and infer new knowledge about the ontology, and access to the Semantic Query-Enhanced Web Rule Language (SQWRL), which supported querying the ontology to retrieve knowledge inferred by the SWRL rules [45].

2.3.1.2. Alerts and ontology-driven alert module development

The SWRL rules were used to generate the prescribing alerts using the assertional knowledge about the relationships between the classes. Users interacted with the antibiotic prescribing alerts using the ontology-driven alert module. The alert module and associated SWRL rules were implemented in Java. The alert module included sections for patients, medication allergies, pseudomonal risk factors that altered the recommendations, antibiotic orders, microbiology culture orders, microorganism and susceptibility results, ‘inline’ prescribing alerts, and history of resistant organisms (the latter were generated by a SWRL rule to display the date and type of previous resistant organisms). Pop-up boxes for adding medication allergies, creating new antibiotic orders and microbiology orders, and website links to internal and external knowledge resources such as the Centers for Disease Control and Prevention, UpToDate, Columbia University Medical Center Infectious Diseases department, and PubMed were also included. Some error checking functionality of required data entry fields and order deletion warnings were supported.

2.3.1.3. Patient database development

A patient database that included de-identified patient microbiology culture results and findings for 81 fictitiously named individuals of the patient class in the ontology was created to evaluate the SWRL rules in generating the prescribing alerts. The patient data were manually entered in Protégé.

2.4. Usefulness evaluation studies

2.4.1. Knowledge management task usefulness study

Two ID experts, who had previously participated in the laddering study, interacted with the ontology using Protégé and the ontology rules using the SWRLTab to perform a series of view (e.g., What are the common mechanisms of antimicrobial resistance listed in the ontology?) and edit (e.g., Modify Rule 1 to find all patients who have a positive culture for gram-negative organisms) tasks that included 20 questions based on the National Guideline Clearinghouse (NGC) bacterial meningitis and NYP ID CAP and UTI guidelines. Each task included multiple questions. Participants received a brief tutorial on Protégé and the SWRLTab prior to beginning the study.

During the session, participants were asked to think aloud as they completed tasks and their verbalizations and screenshots were recorded using Morae™ (TechSmith, Okemos, MI). Actual task complexity was measured by time to complete task, mouse clicks, and keystrokes; verbalizations were thematically analyzed to assess ID experts’ perceptions of usefulness on the ontology for knowledge management.

2.4.2. Alert generation task usefulness study

We conducted 9 individual end-user evaluation sessions with non-ID clinicians (Resident and Fellow Physicians and Nurse Practitioners). During each session, participants were asked to think aloud as they interacted with the ontology-driven alert module to create antibiotic and culture orders based on five patient scenarios that included empiric and directed prescribing tasks. Some participants received an additional directed prescribing task to ensure that they interacted with all of the prescribing alerts. Participants also explored the alert module by creating inappropriate and incomplete orders and adding medication allergies and pseudomonal risk factors to determine the system response. Verbalizations and screenshots were captured using Morae™ as described above.

Afterwards, participants completed an 11-question, 5-point (strongly disagree to strongly agree) survey to evaluate the potential usefulness of the ontology-driven alert module. The audio and Morae™ recordings were analyzed to identify themes related to usefulness and intention to use the alert module. Survey data were summarized with descriptive statistics.

3. Results

3.1. Ontology Development Process

The ontology includes 136 classes, 10 properties, and 1,636 restrictions that represented DL expressions to infer knowledge about the antibiotic agent, microorganism, and ID classes. The salient classes and definitions of the ontology along with BFO class type are displayed in Table 1 and Table 2 presents the properties defined in the ontology. Figure 1 displays a screenshot of the ontology for the Pseudomonas aeruginosa class.

Table 1.

Salient ontology classes, definitions, and BFO class type

Class	Definition	BFO Class Type
Mechanism of Resistance	The methods in which bacteria develop resistance to an antibiotic agent and are able to withstand the effect of the antibiotic agent	Realizable Entity: Disposition
Mechanism of Action	The biochemical interaction through which a drug substance produces its pharmacological effect. In this context, referring to the mechanism of action of antibiotics	Realizable Entity: Function
Antibiotic	A chemotherapeutic agent or substance that kills (microbicidal) or inhibits (microbistatic) the growth of microorganisms such as bacteria and treat a bacterial infection.	Realizable Entity: Role
Antibiotic Classification	Classification used to describe those antibiotics with a similar chemical structure and antibiotics within the same class generally have similar effects.	Realizable Entity: Role
Gram-Negative Bacteria Classification	Classification used to describe those bacteria that do not retain crystal violet dye in the Gram staining protocol (are red).	Independent Continuant: Object
Gram-Positive Bacteria Classification	Classification used to describe those bacteria that are stained dark blue or violet by Gram staining.	Independent Continuant: Object
Microorganism	A microscopic living system that includes bacteria, fungi, archaea, and protists, some microscopic plants and animals such as plankton, and animals such as the planarian and the amoeba.	Independent Continuant: Object
Bacterium	Bacteria are a large group of unicellular microorganisms. Typically a few micrometers in length, bacteria have a wide range of shapes, ranging from spheres to rods and spirals	Independent Continuant: Object
CAP	Community-acquired pneumonia (CAP) is a disease in which individuals who have not recently been hospitalized develop an infection of the lungs (pneumonia). CAP is a common illness and can affect people of all ages. CAP often causes problems like breathing, fever, chest pains, and a cough. CAP occurs because the areas of the lung which absorb oxygen (alveoli) from the atmosphere become filled with fluid and cannot work effectively.	Processual Entity: Process
UTI	A urinary tract infection (UTI) is a bacterial infection that affects any part of the urinary tract.	Processual Entity: Process

Table 2.

Salient ontology properties

Object Property	Domain Class	Range Class
Causes	Bacterium	Clinical Disease
hasAntibioticCoverageBy	Bacterium	Antibiotic
hasMechanismofAction	Antimicrobial	Mechanism of Action
hasMechanismofResistance	Bacterium	Mechanism of Resistance
isAntibioticCoverageFor	Antibiotic	Bacterium
isCausedBy	Clinical Disease	Bacterium
isTreatedWithAlternativeTherapy	Clinical Disease	Antibiotic
isTreatedWithAlternativeTherapyAspiration	Clinical Disease	Antibiotic
isTreatedWithPrimaryTherapy	Clinical Disease	Antibiotic
isTreatedWithPrimaryTherapyAspiration	Clinical Disease	Antibiotic
Treats	Antibiotic	Clinical Disease

Figure 1.

Screenshot of the ontology for the Pseudomonas aeruginosa class

3.2. Evaluation of ontology correctness (Intrinsic evaluation)

3.2.1. Ontology design principles

The ontology adheres to four of the nine relevant Desiderata characteristics: concept orientation, formal definitions, reject “not elsewhere classified,” and context representation. Five characteristics (nonsemantic concept identifiers, polyhierarchy, multiple granularities, multiple consistent views, recognize redundancy) were not satisfied and will be addressed in future iterations.

3.2.2. Domain expert review

No differences were observed between the hierarchies created by the two ID experts using the laddering technique and the ontology. Comparison of the participant hierarchies demonstrated agreement about the ontology classes; there were some minor differences between the examples given by participants, but responses were correct. As a result, it was not necessary to refine the antimicrobial-microorganism ontology further before conducting the extrinsic evaluations.

3.3. Evaluation of ontology usefulness (Extrinsic evaluation)

3.3.1. Development of Artifacts

3.3.1.1. Rules

Twenty-three SWRL rules were written using the SWRLTab to generate the antibiotic prescribing alerts from the assertive relational knowledge in the ontology. Multiple rules were required to represent the antibiotic-microorganism mismatch and the non-recommended empiric antibiotic therapy alerts. An example of the rule for the medication-allergy alert is expressed below and other examples are provided in Appendix B.

• Patient(?patient) ^

• hasAdmissionVisit(?patient, ?visit) ^

• hasAdmitDate(?visit, ?aTime) ^

• temporal:hasTime(?aTime, ?admitDate) ^

• hasMedicationAllergy(?patient, ?medAllergy) ^

• hasInpatientOrder(?visit, ?orderPharm) ^

• Antibiotic_Pharmacy_Order(?orderPharm) ^

• hasAntibioticOrderName(?orderPharm, ?abx) ^

• abox:hasClass(?medAllergy, ?allType) ^

• abox:hasClass(?abx, ?allType) ^

• → sqwrl:select(?patient, ?visit, ?admitDate, ?medAllergy, ?allType, ?abx)

3.3.1.2. Alerts and ontology-driven alert module

Three types of antibiotic prescribing alerts pertaining to the functional requirements were identified through the focus groups and semi-structured interviews and developed. The antibiotic-microorganism mismatch alert was triggered by a new antibiotic order and notified the clinician when the patient had a gram-positive or gram-negative resistant organism to the ordered antibiotic. If the user continued with the order after receiving the pop-up warning alert, a second alert appeared in the ‘inline’ alert section, which persisted as part of the medical record. The medication-allergy alert was triggered by a new antibiotic order and/or a new medication allergy and notified the clinician when the patient had an allergy to the ordered antibiotic. The alert also provided antibiotic-class hierarchical relations checking, e.g., methicillin is a type of penicillin. As described above, the alert consisted of two parts: first, a pop-up warning alert and second, if the user continued with the order, an alert in the ‘inline’ alert section. The non-recommended empiric antibiotic therapy alert was also triggered by a new antibiotic order and notified the clinician when the patient had an empiric diagnosis of CAP or UTI and the ordered antibiotic did not match the recommended local NYP empiric antibiotic therapy guidelines. If the user continued, the antibiotic order was accepted. Figure 2 provides an annotated example of the non-recommended empiric therapy for the UTI alert illustrating how text from the alert message was connected to knowledge in the ontology, represented as classes, properties, and relationships.

Figure 2.

Mapping the non-recommended empiric therapy for UTI alert with the ontology classes, properties, and relationships.

Arrow 1- shows that Azithromycin, (an instance of the Azithromycin class, which is a subclass of the Antibiotic class) is retrieved from the ontology using the hasAntibioticOrderName property

Arrow 2- shows that Rachel Wagner is an instance of the Patient class

Arrow 3- shows that Catheter-Associated UTI (an instance of the Catheter-Associated UTI class, which is a subclass of the UTI class and a subclass of the Clinical Disease class) is retrieved from the ontology using the hasAdmitReason property

Arrow 4- shows that Azithromycin is connected with the Catheter-Associated UTI in the ontology using the treats property

Arrow 5- shows that Penicillin G (an instance of the Penicillin G class, which is a subclass of the Penicillin class and a subclass of the Antibiotic Classification Class) is retrieved from the ontology using the hasMedicationAllergy class

Figure 3 displays a screenshot of the final version of the ontology-driven alert module with pop-ups for a new antibiotic order and the triggered antibiotic-microorganism mismatch alert.

Figure 3.

Ontology-driven alert module with (A) an antibiotic order and (B) and antibiotic-microorganism mismatch alert

3.3.1.3. Patient database

The patient database contained 188 classes, 50 properties, and 2,167 unique instances of classes that pertain to patient demographics and medical history (e.g., gender, medical record number, patient allergies, pseudomonal risk factors), admission visits (e.g., visit ID, admission reason, inpatient bed location, antibiotic orders, microbiology orders, susceptibility results), and timestamp data pertaining to all visits and orders.

3.3.2. Usefulness evaluations

3.3.2.1. Knowledge management task usefulness

It took ID expert #1 41:17 minutes (280 mouse clicks, 159 keystrokes) and ID expert #2 42:36 minutes (413 mouse clicks, 174 keystrokes) to complete the four view and edit tasks (Appendix C). The edit tasks (Task 2 and 4), where users had to manipulate portions of the ontology and rules, took 6:31 minutes longer than the view tasks (Task 1 and 3) (45:12 minutes vs. 38:41 minutes) but required 35 fewer mouse clicks than the view tasks (329 vs. 364). The domain knowledge tasks (Task 1 and 2), where participants were searching for specific information in the ontology or modifying the ontology, took 16:05 minutes longer than the knowledge rule tasks (Task 3 and 4) (49:59 vs. 33:54 minutes).

Three themes emerged about the usefulness of the ontology for representing and maintaining antimicrobial treatment knowledge rules. Participants described the ontology as useful for performing knowledge management tasks. ID expert #2 expressed that the ontology was “useful to get to the end result and help the clinician prescribe antibiotics.” Another reoccurring theme was that the ontology knowledge management tasks were irrelevant to the ID Physician role. ID expert #1 expressed that “ [they] didn’t feel that ID Physician needed to provide that relationship ID knowledge because they have guidelines. Need to apply knowledge as end-user. Perhaps user of system could be someone in an epidemiology department who is trained to use the system and make these update- again not the ID Physician.” Lastly, participants also articulated that the ontology was not easy to use. ID expert #2 stated “Since I’m not familiar with programming, think it was confusing… difficult. I think it’s easy to scroll around until I find something. It’s not difficult; it’s just kind of complex. It’s not hard to do, not intuitive I guess. Not particularly hard to do. A lot of trial and error involved to figure out what’s right until the computer gives feedback— the red screen.”

3.3.2.2. Alert generation task usefulness

Most participants had both 1–5 years of clinical experience and experience using a Computerized Prescribing Order Entry (CPOE) or Electronic Health Record (EHR) system. Four were female and five were male. Participants’ clinical specialty included internal medicine (3), emergency medicine (1), surgery (1), hospitalist (1), anesthesiology (1), psychiatry (1), and intensive care unit (1). Participants created 167 new orders (104 antibiotic and 67 culture orders). On average, each participant created 19 new antibiotic and culture orders. Table 3 presents data for the time to complete the task (minutes) and mouse clicks across clinical roles. All participants perceived the alert module useful for improving the quality of care, appropriately prescribing antibiotics, providing reminders, and supplying relevant information; 8 as useful for enhancing their effectiveness at antibiotic prescribing, decreasing the error potential, presenting additional information, and supplying accurate information; 5 as useful for completing antibiotic prescribing tasks faster. All perceived the individual alerts as useful.

Table 3.

Alert generation task usefulness summary

Task	Resident Physician		Fellow Physician		Nurse Practitioner		Total
	Time to complete	Mouse click	Time to complete	Mouse click	Time to complete	Mouse click	Total time	Total click
1. Empiric Task
1.1 CAP	8:45	105.8	10:33	78.5	7:33	120.0	26:52	304.3
1.2 Catheter-associated	4:55	50.3	3:03	36.5	4:30	63.7	12:28	150.4
UTI
1.3 Complicated UTI	4:45	44.5	4:38	42.5	3:07	35.7	12:29	122.7
Subtotal	18:25	200.5	18:13	157.5	15:10	219.3	51:48	577.3
2. Directed Task
2.1 Antibiotic-Microorganism mismatch	3:45	68.5	5:30	59.5	4:59	78.0	14:14	206.0
2.2 Antibiotic-Microorganism mismatch	2:59	90.5	3:53	27.5	2:43	37.5	9:36	155.5
2.3 Antibiotic-Microorganism mismatch	6:17	83.0	8:56	221.0	5:38	103.0	20:51	407.0
Subtotal	13:02	242.0	18:18	308.0	13:21	218.5	44:41	768.5
Total	31:26	442.5	36:31	465.5	28:30	437.8	1:36:28	1345.8

Two themes emerged about the usefulness of the ontology for generating alerts to provide feedback to clinicians during antibiotic prescribing.

Usefulness for enhancing patient safety:

• “Ok, ‘You're ordering erythromycin.’ Did I not look at his…I didn't look at his allergies and I ignored it! Ok, very good.” Nurse Practitioner, Non-recommended Therapy for CAP Alert

• “It would've caught the levo [floxacin] and gent [amicin]. Right, because it's already counting for the [penicillin] allergy too, so in a way actually, they sort of blanket, putting it all together. Actually I think it's handy, because that's like oh, I might've not even have noticed that the first time, which honestly I didn't. And it caught it and it's suggesting actually levo [floxacin] and gent [amicin] for this guy, which is handy.” Resident Physician, Non-recommended Therapy for UTI Alert

Usefulness for antibiotic prescribing:

• “Because I think it does help, because not everybody will remember. Like for instance, I don't always treat UTIs, I just kind of go with what bugs often might pop up. So it's good to be reminded of what the guidelines are.” Fellow Physician, Non-recommended Therapy for UTI Alert

• “I think this is really helpful, especially for somebody who is new like me. I like the alerts, because I think it's a good idea. It's flash-full; it's a bit of a warning to me like, ‘Hello?’” Nurse Practitioner, Antibiotic-Microorganism Mismatch Alert

4. Discussion

This paper describes the development and evaluation of an antimicrobial-microorganism ontology using formal ontology methods that can be reused to support the development and evaluation of other ontologies. In developing our ontology, we made several design decisions pertaining to the use of existing ontologies, upper ontologies, and how the ontology was represented that best fit the purpose and scope of our ontology. Our decision to include an upper ontology in our application ontology was worthwhile as the additional structure provided by BFO challenged us in how the ontology classes and properties were logically designed and defined; creating an ontology that can be integrated into other ontologies and reused to support additional functionality. We also developed several artifacts in order to extrinsically evaluate the ontology, including 23 prescribing rules. The choice to represent our rules in SWRL was guided by SWRL’s tight integration with OWL and because it was a proposed W3C standard at the time of development. Representing some of the rules required complex workarounds, however, with the advancement of the Rule Interchange Format (RIF), transforming the SWRL rules into RIF-Basic Logic Dialogue (BLD) would alleviate some of the challenges we experienced, improve the efficiency of the rules, and enhance the functionality of the alert module.

In evaluating the correctness of the ontology, we found that it satisfied four of the 12 characteristics identified in Cimino’s Desiderata, although only nine characteristics (concept orientation, formal definitions, reject “not elsewhere classified,” context representation, nonsemantic concept identifiers, polyhierarchy, multiple granularities, multiple consistent views, and recognize redundancy) were relevant to the evaluation given its recentness. As the ontology is further refined, improvements will be made to address those characteristics that were not satisfied to further strengthen the ontology structure and content. Assessing the usefulness of the ontology for performing knowledge management tasks revealed that the ID experts perceived it as useful for maintaining the ontology, but not easy to use. Even though BFO enhanced the ontology design, despite definitions and examples, participants experienced some obstacles in conceptualizing the BFO hierarchy and navigating the ontology. While the knowledge rules tasks were considered to be more difficult because of the level of interaction required with the SWRLTab, the domain knowledge tasks where participants had to negotiate through the ontology and the BFO hierarchy to find specific classes required the most time to complete. Further exploration is needed to identify whether an alternative upper ontology would fit our development needs, while also providing non-ontologists with a more familiar hierarchy.

Measuring the usefulness of the ontology for alert generation demonstrated that the clinicians perceived the ontology-driven alert module as useful for guiding antibiotic prescribing. Although all participants expressed that the alerts were useful and that they would be likely to use the alert module, further investigation is needed to determine if improvements in the functionality would have an effect on the perception that the alert module is useful for completing antibiotic prescribing tasks faster.

4.1. Significance of the study

This study contributed a novel methodological approach to intrinsically and extrinsically evaluate ontology quality. First, utilizing the laddering technique to measure the correctness of the antimicrobial-microorganism ontology based on domain expert agreement was a unique approach. Several studies have discussed and demonstrated the usefulness of the laddering technique to elicit domain knowledge to generate a new hierarchy of knowledge [41, 42]; however, instead of using the laddering technique to create the ontology, we conducted the laddering technique with ID experts to validate a subset of the ontology for correctness. This method was efficient and ensured that the ontology was reflective of the domain and not the developer’s viewpoint. Second, we introduced a user-centered approach to evaluate the usefulness of the ontology for knowledge management with domain experts as opposed to ontologists or ontology engineers; demonstrating the utility of incorporating domain expert feedback in the evaluation phase to ensure correctness as a complementary method to automated and manual auditing techniques [46–51]. Although the ID experts expressed that it was not easy to interact with the ontology and that performing knowledge management tasks was not appropriate for experts who also have a clinical role; this still provided a wealth of feedback and was a beneficial step in evaluating the usefulness of the ontology. Third, we developed three artifacts to test the usefulness of the ontology for guiding antibiotic prescribing. This evaluation study highlighted the possibilities of generating ontology-based alerts in a graphical-user interface to provide feedback to clinicians during antibiotic prescribing in a useful manner.

4.2. Limitations of the study

The primary limitation of this study is the limited generalizability of the evaluation findings due to the relatively small sample size, limited ontology scope, and single academic medical center setting with clinicians who may not be representative of the broader population. Additionally, the evaluations occurred in a laboratory setting, thus perceptions of usefulness of the ontology to generate prescribing alerts might vary from data collected in a clinical setting. However, the approaches that we applied support the extensibility of the ontology for reuse in other settings and circumstances. The decision to use the Sanford Guide to Antimicrobial Therapy based on content coverage also limits the interoperability of the ontology, and licenses would be needed to facilitate sharing of this ontology because of its use of the content from this reference.

5. Conclusions

This study contributed to the understanding of ontology development and evaluation methods and addressed one knowledge gap related to using ontologies as a CDS system component: a need for formal ontology evaluation methods to measure the quality of ontologies from the perspective of their intrinsic characteristics and usefulness for specific tasks. In addition to implementing formal methods to develop ontologies, formally evaluating ontologies is an important step to produce ontologies that have the potential to be reused and interoperable with existing ontologies. Antimicrobial resistance is a significant global health problem, thus we developed and intrinsically and extrinsically evaluated an antimicrobial-microorganism ontology to demonstrate a practical solution that addresses the need for judicious antibiotic prescribing for empirically treating CAP and UTIs and for the directed treatment of organisms from blood, respiratory, and urine cultures. Our approach ensured that the ontology was correctly structured and that the content accurately reflected domain knowledge. Additionally, our methods demonstrated the usefulness of our user-centered approach to evaluate the quality of ontologies from the perspective of potential system users regarding of usefulness of the ontology for a specific task. These methods can be applied to develop and evaluate additional ontologies, and the findings of this study provide evidence of the potential usefulness of this ontology for both knowledge maintenance and alert generation.