Abstract
When coupled with a common information model, a common terminology for clinical decision support (CDS) and electronic clinical quality measurement (eCQM) could greatly facilitate the distributed development and sharing of CDS and eCQM knowledge resources. To enable such scalable knowledge authoring and sharing, we systematically developed an extensible and standards-based terminology for CDS and eCQM in the context of the HL7 Virtual Medical Record (vMR) information model. The development of this terminology entailed three steps: (1) systematic, physician-curated concept identification from sources such as the Health Information Technology Standards Panel (HITSP) and the SNOMED-CT CORE problem list; (2) concept de-duplication leveraging the Unified Medical Language System (UMLS) MetaMap and Metathesaurus; and (3) systematic concept naming using standard terminologies and heuristic algorithms. This process generated 3,046 concepts spanning 68 domains. Evaluation against representative CDS and eCQM resources revealed approximately 50–70% concept coverage, indicating the need for continued expansion of the terminology.
Introduction
Need for data standardization
Despite the demonstrated potential for clinical decision support (CDS) to improve care quality and promote patient safety (1–4), CDS availability continues to be limited in most clinical settings (5–7). An important reason for this limited CDS availability is the difficulty of scaling CDS across institutions (8–10), with the lack of data standardization being a predominant barrier to sharing (11). Electronic clinical quality measurement (eCQM), which shares many requirements with CDS and can be implemented using a common underlying system (12), has a similar need for standardized data. Indeed, the U.S. Office of the National Coordinator for Health IT and the Centers for Medicare & Medicaid Services are sponsoring an initiative known as the Clinical Quality Framework to develop a harmonized set of standards to fulfill the needs of both CDS and eCQM (13).
Figure 1 provides an overview of aspects of data standardization for CDS and eCQM. One aspect of standardization is the information model, which identifies data classes (e.g., Problem), attributes (e.g., problem code), and the relationship of classes to one another (e.g., the relationship of Problems to Encounters; not shown). Coded attributes describe concepts such as “diabetes mellitus,” which in turn may be defined by a value set of instance codes that are indicative of the concept (e.g., SNOMED-CT 314902007, type II diabetes mellitus with peripheral angiopathy).
Figure 1.
Conceptual framework.
Need for a concept terminology for CDS and eCQM
Data standardization efforts in CDS and eCQM have generally focused on standardization of (i) the information model, (ii) the superset of instance codes that may be used within coded attributes, and, in some cases, (iii) individual value sets (14, 15) However, to our knowledge, there has been no systematic effort to define a common concept terminology for CDS and eCQM to facilitate knowledge sharing and semantic interoperability.
Many standard terminologies, such as SNOMED-CT, RxNorm and LOINC, are available for use in CDS and eCQM with relatively adequate breadth, depth, and granularity (16). However, the sheer volume of concepts in these terminologies can make it challenging to ensure that different CDS and eCQM implementers choose the same concepts in their respective implementations. For example, the number of coded concepts in the UMLS Metathesaurus (>1,400,000), SNOMED-CT (> 310,000), RxNorm (> 93,000), LOINC (> 46,000), and ICD-10 (> 12,000) makes consistent concept selection challenging (17). Meanwhile, many terminologies remain semantically incompatible (18). The diversity in different terminological systems hampers the possibility of sharing and reasoning with data within different systems (11). Therefore, the challenge lies less with the lack of relevant standards, but more with the fact that multiple terminologies are in concurrent use (18), and with the sheer volume of concepts. Furthermore, the lack of hierarchical structures in some terminologies makes it difficult to find useful terms that are less specific, as is often needed for CDS and eCQM. Consequently, it is imperative to identify and maintain a much smaller subset of broader concepts with utility for computerized CDS and eCQM. Here, we describe an effort to meet this need within the context of OpenCDS, which is a multi-institutional collaborative initiative to develop open-source, standards-based tools and resources to enable CDS and eCQM at scale (12, 19).
Methods
Project context and operational use of terminology
The concept terminology was developed in the context of the OpenCDS effort to support CDS and eCQM. OpenCDS has been implemented in a number of electronic health record (EHR) systems and provides a reference implementation of the HL7 vMR data model standard (12, 19–22). The vMR was designed originally for CDS but has been subsequently applied to eCQM as well (12). The vMR contains 68 coded attributes, such as Adverse Event, Encounter Type, Goal Focus, Observation Focus, Problem, Procedure, Medication, and Supply (Table 1).
Table 1.
Summary of coded attributes in the vMR information model
| Entities: |
| Terms Relating to All Entities |
| Entity Type |
| Entity Relationship |
| Person |
| Ethnicity |
| Gender |
| Preferred Language |
| Race |
| Substance |
| Manufacturer |
| Medication |
| Medication Branded |
| Medication Generic |
| Substance Form |
| Supply |
| Supply |
| Clinical Statements: |
| Terms Relating to All Clinical Statements |
| Clinical Statement Relationship |
| Data Source Type |
| Adverse Event |
| Adverse Event |
| Adverse Event Affected Body Site |
| Adverse Event Affected Body Site Laterality |
| Adverse Event Agent |
| Adverse Event Criticality |
| Adverse Event Severity |
| Adverse Event Status |
| Encounter |
| Encounter Type |
| Encounter Criticality |
| Appointment Proposal Criticality |
| Appointment Request Criticality |
| Goal |
| Goal Criticality |
| Goal Focus |
| Goal Status |
| Goal Target Body Site |
| Goal Target Body Site Laterality |
| Observation |
| Observation Coded Value |
| Observation Criticality |
| Observation Focus |
| Observation Interpretation |
| Observation Method |
| Observation Target Body Site |
| Observation Target Body Site Laterality |
| Observation Unconducted Reason |
| Problem |
| Problem |
| Problem Affected Body Site |
| Problem Affected Body Site Laterality |
| Problem Importance |
| Problem Severity |
| Problem Status |
| Procedure |
| Procedure |
| Procedure Approach Body Site |
| Procedure Approach Body Site Laterality |
| Procedure Criticality |
| Procedure Method |
| Procedure Target Body Site |
| Procedure Target Body Site Laterality |
| Unconducted Procedure Reason |
| Substance Administration |
| Dose Type |
| Dosing SIG |
| Information Attestation Type |
| Substance Administration Approach Body Site |
| Substance Administration Approach Body Site Laterality |
| Substance Administration Criticality |
| Substance Administration General Purpose |
| Substance Administration Target Body Site |
| Substance Administration Target Body Site Laterality |
| Substance Delivery Method |
| Substance Delivery Route |
| Undelivered Substance Reason |
| Supply |
| Supply Criticality |
| Supply Target Body Site |
| Supply Target Body Site Laterality |
| Undelivered Supply Reason |
In OpenCDS, CDS or eCQM modules are authored as a series of human-readable rules and then translated into machine-executable knowledge. Concepts are accessed via drop-down lists specific to the type of concept involved (e.g., gender) (Fig. 2). These concepts, in turn, are mapped to value sets containing applicable local or standard codes. The use of concepts enables a clear separation of concerns between terminology mapping and logic authoring.
Figure 2.
Use of concept terminology in OpenCDS knowledge authoring.
Objectives and requirements
The objectives of this project were to (i) define a standard and extensible approach for curating a concept terminology for CDS and eCQM that can be leveraged by the OpenCDS community and to (ii) populate the terminology with an initial set of common, high-level concepts useful for CDS and eCQM knowledge authoring. Requirements included (i) adherence to the 80-20 rule, with a goal of initial inclusion of high coverage of concepts likely to be needed for typical CDS or eCQM use cases; (ii) the leveraging of standard terminologies; (iii) internal consistency; and (iv) avoidance of duplicate concepts.
Overview of approach
The terminology was developed using three steps. First, relevant concepts were identified in a systematic, physician-curated manner. Second, concepts were de-duplicated using UMLS MetaMap and Metathesaurus. Finally, concepts were named using standard terminologies and heuristic algorithms. These steps are outlined in greater detail below.
Step 1a: Candidate concept identification
To identify relevant concepts, we first reviewed the Healthcare Information Technology Standards Panel (HITSP)’s Clinical Document and Message Terminology specification (HITSP C80, Version 2.0.1) (23). This document defines the vocabularies used by HITSP specifications for clinical documents and messages to support the interoperable transmission of information. If this specification defined a finite value set for a targeted coded attribute type, that value set was used as the set of candidate concepts for physician review and curation in the next phase of this step.
If HITSP C80 did not define a value set for a coded attribute type, we next searched the Public Health Information Network Vocabulary Access and Distribution System (PHIN VADS) and National Library of Medicine’s Value Set Authority Center (VSAC). If an appropriate value set was identified here, then that value set was identified as the candidate concept set.
If the above resources did not identify a relevant value set, or if the value set identified was extremely large in scope, the potential concepts for physician review were restricted using various methods. For example, HITSP C80 recommends concepts using the Veteran Administration and Kaiser Permanente (VA/KP) problem list subset of SNOMED-CT for describing problems (23). However, the VA/KP problem list subset contains over 15,000 concepts, making it a challenge to review. Therefore, we instead used the Clinical Observations Recording and Encoding (CORE) subset of SNOMED-CT (24). CORE was based on datasets submitted by 8 institutions, and it is a frequency-based approach to problem list development. Compared to VA/KP, CORE is smaller, and 94.8% of coded problem entries from Brigham and Women’s Hospital are in the CORE subset (4), indicating high coverage of used concepts. For our purposes, we started with the 266 CORE problem list entries that were reported by most (8 or 7) of the institutions as the candidate set of problem concepts for potential inclusion in the initial CDS/eCQM terminology. Similar methods were used for enriching the candidate set of concepts designated for physician review. For example, laboratory test concepts were restricted to the LOINC Universal Laboratory Order Codes, whose approximately 300 codes cover more than 95% of the lab test orders in the United States (25).
Step 1b: Physician curation
After candidate concepts were identified in the step above, a physician informaticist (VK) who is a practicing hospitalist reviewed each concept in the candidate set and identified those that have a reasonable likelihood of being useful for CDS purposes based on personal experience. The physician informaticist classified these concepts into 4 categories: 1 - high priority, 2 - moderate priority, 3 - low priority, and 4 - not appropriate. Concepts with priority 1 and 2 were uploaded into the Apelon Distributed Terminology System (DTS) terminology server for management.
Step 2: De-duplication
Duplicate entries are a common problem in terminologies (6, 26), even in the UMLS Metathesaurus (27). To identify and deprecate duplicate concepts, we implemented a systematic methodology for identifying potential duplicates, which were verified through physician review (Fig. 3).
Figure 3.
Strategy to identify duplicate concepts.
Before starting, we identified candidate concepts for de-duplication by excluding concepts that had previously been deprecated or were being used for administrative purposes (e.g., to name a specific quality measure, such as HEDIS Breast Cancer Screening). We then searched for exact string matches to SNOMED-CT terms (including synonyms) in the UMLS Metathesaurus, capturing the corresponding UMLS concept unique identifier (CUI) through the process. This subset of the terminology (T1) represented a large portion of the original set, indicating that SNOMED-CT was a reasonable source for concept names. The remaining OpenCDS concepts (R1) were then screened for perfect string matches to the terms in any other standard terminologies in UMLS, and the results were processed similarly (T2).
For the remaining concepts with no perfect string matches available (R2), the UMLS MetaMap tool (28) was used to identify potential matching UMLS CUIs. Concepts that could not be matched to any UMLS terms in this manner (R3) were generally unique concepts used as intermediate conclusions (e.g., “age >= 50 and < 75 years”) and were not processed further for de-duplication.
Next, we combined all the concepts and the corresponding UMLS CUIs from T1, T2 and T3 and identified potential duplicate concepts sharing the same CUIs. These potential duplicates were reviewed by a physician. If two or more concepts shared the same CUI but were deemed to be distinct, we updated the CUI for one of the concepts using the UMLS Metathesaurus. If two or more concepts were deemed to be duplicative, one was kept and the rest were deprecated.
Step 3: Concept naming
For concepts matched to more than one CUI, the preferred term for each CUI was obtained from the UMLS Metathesaurus and reviewed to identify the most appropriate CUI for the concept. Next, using the CUI associated with each concept, preferred terms from appropriate standard terminologies were obtained by leveraging the UMLS as shown in Fig. 4. Finally, concept names were post-processed for consistency using heuristic algorithms. For example, capitalization schemes were standardized. Also, concepts were named as the first major category followed by any modifiers to facilitate finding all variations on a root concept in a drop-down list. For example, “Bilateral Mastectomy” was renamed “Mastectomy, Bilateral” and “Lower Extremity Amputation” was renamed “Amputation, Lower Extremity.”
Figure 4.
Strategy to identify standard terms for local terminology.
Evaluation of concept coverage
We evaluated the degree of coverage of the concept terminology for sample CDS and eCQM knowledge resources. For CDS, we reviewed the data sections of example Arden Syntax Medical Logical Modules provided in the appendix of version 2.8 of the standard (29). For eCQM, we reviewed the first 50 value sets in the National Quality Forum’s eCQMs from 2011 (30).
Results
Concept identification, de-duplication, and naming
3886 concepts spanning the 68 vMR coded attributes were identified for potential inclusion in the terminology. Following physician informaticist review, approximately 2,200 clinical concepts were selected for inclusion.
Our systematic de-duplication method identified 110 potential duplicates. After review by a physician, 72 concepts were confirmed to be duplicates and deprecated. For example, “Urinary retention” and “Retention of urine” were found to be duplicates, leading to one of the concepts being deprecated. Finally, 1,928 concepts with UMLS CUIs were named using SNOMED-CT, RxNorm, and other standard terminologies included in the UMLS.
Concept upload to Apelon DTS terminology server
Concepts were then uploaded into the Apelon DTS terminology server and mapped to corresponding coded attribute types. Each of these concepts from the external terminologies then became a unique Apelon DTS concept, and a code was assigned automatically to the concept. All concept names were capitalized (proper case). This import also updated the hierarchical relationships so that concepts are the descendants of corresponding vMR coded attribute types. In some cases, a single concept can be associated with two or more coded attribute types, but concepts were defined only once. For example, “Pregnancy Test” can be either an Observation Focus with a possible result value or simply a Procedure that was performed. Accessing “Pregnancy Test” from Observation Focus or Procedure in OpenCDS will bring the user the same concept and code (C1693).
Concept coverage
The terminology created was evaluated against a previously unseen set of Arden Syntax Medical Logic Modules and National Quality Forum eCQMs. This analysis showed that the terminology developed covered approximately 70% of the concepts referenced in the Medical Logic Modules and approximately 50% of the concepts referenced in the eCQMs. Many of the concepts that were not covered by the terminology consisted of concepts for specific medications.
Discussion
Implementing CDS capabilities usually requires several terminologies due to their different domain(s) of coverage and granularity (20, 31). As a result, concurrent use of different terminologies is a significant challenge, and a CDS resource designed for use in one setting may not be readily usable in another setting that uses a different set of terminologies, even when similar concepts are being captured (18). Furthermore, when different institutions use different subsets with non-overlapping terms, significant interoperability challenges occur (4). To address these issues, we built a terminology for CDS and eCQM based on the HL7 vMR information model as a part of the OpenCDS initiative. A central part of this terminology development effort was the definition and application of systematic approaches to de-duplication and naming standardization.
Achieving semantic interoperability for CDS and eCQM depends on the use of common information models and common associated concepts (32). In our study, we sought to define a “starter set” of concepts that have a reasonable likelihood of being useful for CDS or eCQM purposes. However, identifying what terminology is “best” or which term is “common” is challenging. For example, some concepts that are common to ambulatory care may not be relevant in an inpatient scenario. Thus, besides the domain-specific expertise from our group, our strategy was to leverage established sources of common concepts such as the SNOMED-CT CORE problem list and HITSP recommendations.
Implementing effective vocabulary control in medical informatics includes facilitating the selection of the most appropriate concept for a given clinical scenario (33). Wright et al. used human immunodeficiency virus (HIV) as an example to demonstrate the importance of screening the concepts for a specific data attribute. In this example, they note that SNOMED-CT has 138 concepts related to HIV and that, without appropriate filtering, a clinician may easily select an incorrect code by mistake (4). We believe our terminology consisting of common and relevant concepts will lead to less chances for inadvertent selections of inappropriate concepts during knowledge authoring.
We included concepts from standard terminologies, such as SNOMED-CT, LOINC, and HL7. In addition, we have added concepts without correspondence to standard terms, primarily administrative concepts such as intermediate conclusions (e.g., “Denominator Inclusion Criteria Met”) or quality measure specifications (e.g., “HEDIS Frequency of Prenatal Care Measure”). Thus, the OpenCDS terminology brings concepts together from disparate controlled terminologies and non-standard terminologies into a single conceptual dictionary of medical concepts. This approach is supported by the vMR information model, which can make use of both standard and local codes. Although OpenCDS can make use of data expressed in many different medical terminologies, it does so through the use of OpenCDS concepts, which map one or more specific and concrete codes from standard or proprietary medical terminologies to a single OpenCDS concept code. An OpenCDS concept is the interface between the clinical ideas and the data details that represent instantiations of the clinical concepts. The clinical rules use OpenCDS concepts in preference to references to the raw data, and terminology mappings provide implementations of those concepts as value sets of codes from one or more code systems. This separates the logic of the rules from the details of the data that the rules work on, thereby facilitating knowledge authoring, maintenance, and sharing.
We learned some lessons when building and maintaining the foundational terminology. When adding concepts in the future, a careful analysis is needed to determine if a concept closely relates to an existing concept. Care should also be taken to ensure consistent naming schemes (34). To avoid ambiguity and to offer an easy way to identify duplicates during maintenance, full names should be provided, either directly or as a concept property.
Building this terminology is an ongoing task. As identified in the evaluation, while the terminology had substantial coverage of relevant concepts, there were still significant gaps in the content. To address this need for continual enhancement and maintenance, we are developing standard operating procedures for adding new content in a systematic, consistent, and non-duplicative manner. In particular, we are seeking to make it easier for OpenCDS users who are not a part of the core development team to request the addition of new concepts.
While the focus of this study was on CDS and eCQM, there are potentially many other contexts in which a standards-based, domain-optimized terminology could provide value. For example, clinical specialties may wish to define common concepts used in their domains, and information systems may benefit from the definition of common concepts used in specific sub-systems (e.g., order entry). While we have not yet applied our methodology to the development of any additional terminologies, we speculate that the methods used in this study – entailing systematic concept identification, deduplication, and naming – could be helpful in developing and maintaining such domain-optimized terminologies.
Conclusions
In this paper, we shared our experiences in building and maintaining a terminology for CDS and eCQM, which is in active use within the OpenCDS community. To the best of our knowledge, this is the first terminology developed specifically to meet CDS and eCQM needs. These concepts and tools are freely available to the open-source community to use and adapt. Because this terminology was built in reference to a standard HL7 clinical information model, our methods and results are likely to be applicable for implementations in other institutions and settings. We believe our experiences described herein will also be informative for others who seek to maintain controlled terminologies in pursuit of semantic interoperability.
Acknowledgments
The project was supported by NLM training grant (T15LM007124) and the University of Utah Knowledge Management and Mobilization initiative. We thank Bruce E Bray, Phillip B Warner, Tyler J Tippetts, Polina V Kukhareva, and Alisha Edison-Hair for their assistance and fruitful discussion, Ryan Butcher for terminology domain-specific suggestions, and Jack Bowie from Apelon for answering technical questions. KK is currently or recently served as a consultant on CDS to the Office of the National Coordinator for Health IT, ARUP Laboratories, McKesson InterQual, ESAC, Inc., JBS International, Inc., Inflexxion, Inc., Intelligent Automation, Inc., Partners HealthCare, the RAND Corporation, and Mayo Clinic. KK receives royalties for a Duke University-owned CDS technology for infectious disease management known as CustomID that he helped develop. KK was formerly a consultant for Religent, Inc. and a co-owner and consultant for Clinica Software, Inc., both of which provide commercial CDS services, including through use of a CDS technology known as SEBASTIAN that KK developed. KK no longer has a financial relationship with either Religent or Clinica Software. KK has no competing interests related to OpenCDS, which is freely available to the community as an open-source resource. VK is a part owner of Coresys Infotech, a CDS company. All other authors declare no conflict of interest.
References
- 1.Hunt DL, Haynes RB, Hanna SE, Smith K. Effects of computer-based clinical decision support systems on physician performance and patient outcomes: a systematic review. JAMA. 1998;280:1339–46. doi: 10.1001/jama.280.15.1339. [DOI] [PubMed] [Google Scholar]
- 2.Sittig DF, Teich JM, Osheroff JA, Singh H. Improving clinical quality indicators through electronic health records: it takes more than just a reminder. Pediatrics. 2009;124:375–7. doi: 10.1542/peds.2009-0339. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Fillmore CL, Bray BE, Kawamoto K. Systematic review of clinical decision support interventions with potential for inpatient cost reduction. BMC Med Inform Decis Mak. 2013;13:135. doi: 10.1186/1472-6947-13-135. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Wright A, Feblowitz J, McCoy AB, Sittig DF. Comparative analysis of the VA/Kaiser and NLM CORE problem subsets: an empirical study based on problem frequency. AMIA Annu Symp Proc. 2011:1532–40. [PMC free article] [PubMed] [Google Scholar]
- 5.Rector AL, Solomon WD, Nowlan WA, Rush TW, Zanstra PE, Claassen WM. A Terminology Server for medical language and medical information systems. Methods Inf Med. 1995;34:147–57. [PubMed] [Google Scholar]
- 6.Dieng-Kuntz R, Minier D, Ruzicka M, Corby F, Corby O, Alamarguy L. Building and using a medical ontology for knowledge management and cooperative work in a health care network. Comput Biol Med. 2006;36:871–92. doi: 10.1016/j.compbiomed.2005.04.015. [DOI] [PubMed] [Google Scholar]
- 7.Batool R, Khattak AM, Kim TS, Lee S. Automatic extraction and mapping of discharge summary’s concepts into SNOMED CT; Conference proceedings: Annual International Conference of the IEEE Engineering in Medicine and Biology Society; 2013. pp. 4195–8. [Google Scholar]
- 8.Osheroff JA, Teich JM, Middleton B, Steen EB, Wright A, Detmer DE. A roadmap for national action on clinical decision support. Journal of the American Medical Informatics Association. 2007;14:141–5. doi: 10.1197/jamia.M2334. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Simon SR, Kaushal R, Cleary PD, Jenter CA, Volk LA, Orav EJ, et al. Physicians and electronic health records: a statewide survey. Arch Intern Med. 2007;167:507–12. doi: 10.1001/archinte.167.5.507. [DOI] [PubMed] [Google Scholar]
- 10.Sittig DF, Wright A, Osheroff JA, Middleton B, Teich JM, Ash JS, et al. Grand challenges in clinical decision support. J Biomed Informa. 2008;41:387–92. doi: 10.1016/j.jbi.2007.09.003. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Ahmadian L, van Engen-Verheul M, Bakhshi-Raiez F, Peek N, Cornet R, de Keizer NF. The role of standardized data and terminological systems in computerized clinical decision support systems: literature review and survey. Int J Med Inform. 2011;80:81–93. doi: 10.1016/j.ijmedinf.2010.11.006. [DOI] [PubMed] [Google Scholar]
- 12.Kukhareva P KK, Shields DE, Barfuss DT, Halley AM, Tippetts TJ, et al. Clinical Decision Support-based Quality Measurement (CDS-QM) framework: prototype implementation, evaluation, and future directions. AMIA Annu Symp Proc. 2014:825–34. [PMC free article] [PubMed] [Google Scholar]
- 13.Clinical Quality Framework Initiative. [cited 2015 March 9]. Available from: http://wiki.siframework.org/Clinical+Quality+Framework+Initiative.
- 14.Clinical Quality Framework. [cited 2015 March 9]. Available from: http://wiki.siframework.org/file/view/20150205+CQF+SI+All+Hands+FINAL.pdf.
- 15.National Quality Forum [cited 2015 March 9]. Available from: http://www.qualityforum.org/projects/e-g/emeasures/electronic_quality_measures.aspx.
- 16.Cimino JJ, Zhu X. The practical impact of ontologies on biomedical informatics. Yearb Med Inform. 2006:124–35. [PubMed] [Google Scholar]
- 17.Bodenreider O. Biomedical ontologies in action: role in knowledge management, data integration and decision support. Yearb Med Inform. 2008:67–79. [PMC free article] [PubMed] [Google Scholar]
- 18.Kawamoto K, Del Fiol G, Lobach DF, Jenders RA. Standards for scalable clinical decision support: need, current and emerging standards, gaps, and proposal for progress. Open Med Inform J. 2010;4:235–44. doi: 10.2174/1874431101004010235. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.OpenCDS [cited 2015 March 9]. Available from: www.opencds.org.
- 20.Nachimuthu SK, Lau LM. Applying hybrid algorithms for text matching to automated biomedical vocabulary mapping. AMIA Annu Symp Proc. 2005:555–9. [PMC free article] [PubMed] [Google Scholar]
- 21.Suralik M, Arzt NH, Chertcoff D, Austin R. The Immunization Calculation Engine: open source clinical decision support for immunizations. J Healthc Inf Manag. 2013;27:32–7. [Google Scholar]
- 22.Virtual Medical Record (vMR) for Clinical Decision Support. [cited 2015 March 9]. Available from: http://wiki.hl7.org/images/7/71/HL7vMR_vMR_Domain_Analysis_Model_Release_1.pdf.
- 23.HITSP C 80 - Clinical Document and Message Terminology Component. [cited 2015 March 9]. Available from: http://www.hitsp.org/ConstructSet_Details.aspx?&PrefixAlpha=4&PrefixNumeric=80.
- 24.The CORE Problem List Subset of SNOMED CT. [cited 2015 March 10]. Available from: http://www.nlm.nih.gov/research/umls/Snomed/core_subset.html.
- 25.Universal Laboratory Order Codes from LOINC. [cited 2015 March 9]. Available from: https://loinc.org/downloads/usage/orders.
- 26.Baorto D, Li L, Cimino JJ. Practical experience with the maintenance and auditing of a large medical ontology. J Biomed Informa. 2009;42:494–503. doi: 10.1016/j.jbi.2009.03.005. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 27.Huang KC, Geller J, Elhanan G, Perl Y, Halper M. Auditing SNOMED Integration into the UMLS for Duplicate Concepts. AMIA Annu Symp Proc. 2010:321–5. [PMC free article] [PubMed] [Google Scholar]
- 28.Interactive MetaMap. [cited 2015 March 9]. Available from: http://ii.nlm.nih.gov/Interactive/UTS_Required/metamap.shtml.
- 29.Arden Syntax v2.8 (Health Level Seven Arden Syntax for Medical Logic Systems V. [cited 2015 March 9]. Available from: http://www.hl7.org/implement/standards/product_brief.cfm?product_id=268.
- 30.National Quality Forum Electronic Quality Measures (eMeasures) [cited 2015 July 5]. Available from: http://www.qualityforum.org/Projects/e-g/eMeasures/Electronic_Quality_Measures.aspx.
- 31.Kawamoto K, Lobach DF. Design, implementation, use, and preliminary evaluation of an UMLS-enabled terminology Web service for clinical decision support. AMIA Annu Symp Proc. 2006:979. [PMC free article] [PubMed] [Google Scholar]
- 32.Oemig F, Blobel B. Semantic interoperability adheres to proper models and code systems. A detailed examination of different approaches for score systems. Methods Inf Med. 2010;49:148–55. doi: 10.3414/ME9304. [DOI] [PubMed] [Google Scholar]
- 33.Gambarte ML, Osornio AL, Martinez M, Reynoso G, Luna D, de Quiros FG. A practical approach to advanced terminology services in health information systems. Stud Health Technol Inform. 2007;129:621–5. [PubMed] [Google Scholar]
- 34.Chute CG, Elkin PL, Sherertz DD, Tuttle MS. Desiderata for a clinical terminology server. AMIA Annu Symp Proc. 1999:42–6. [PMC free article] [PubMed] [Google Scholar]