Abstract
We developed a method to evaluate the extent to which the International Classification of Function, Disability, and Health (ICF) and SNOMED CT cover concepts used in the disability listing criteria of the U.S. Social Security Administration’s “Blue Book.” First we decomposed the criteria into their constituent concepts and relationships. We defined different types of mappings and manually mapped the recognized concepts and relationships to either ICF or SNOMED CT. We defined various metrics for measuring the coverage of each terminology, taking into account the effects of inexact matches and frequency of occurrence. We validated our method by mapping the terms in the disability criteria of Adult Listings, Chapter 12 (Mental Disorders). SNOMED CT dominates ICF in almost all the metrics that we have computed. The method is applicable for determining any terminology’s coverage of eligibility criteria.
Introduction
Under Title II of the Social Security Act, the U.S. Social Security Administration (SSA) provides Disability Insurance (DI) benefits for workers. Under Title XVI of the Act, the SSA administers Supplemental Security Income (SSI) payments for the poor, aged, and disabled with no or little work experience. Each year, SSA renders decisions on millions of cases at the initial and subsequent reconsideration and appeal levels. To determine whether an adult is disabled, SSA’s regulations provide a five-step sequential evaluation process, based on the statutory definition of disability.1 The third step in the sequential evaluation process for an adult claimant is the “listing level”, where an SSA claim adjudicator determines whether the severity of a claimant’s impairment meets or medically equals a specific listing and whether the impairment meets a duration requirement. The listings are enumerated and organized into organ-system–dependent chapters in SSA’s “Blue Book,”2 which has separate parts for adult and childhood listings. Figure 1 shows a partial view of the Blue Book’s chapter on adult mental disorders.
Figure 1.
A view of the SSA Blue Book Adult Mental Disorders Chapter. Each section of the chapter contains detailed criteria for concluding that a claimant meets the definition of the disorder.
At present time the evaluation of disabilities is largely a document-driven process, where only the International Classification of Diseases (ICD) is used to code data relating to diseases. If more structured data documentation can be introduced into the process, it will have enormous impact on the ability to expedite the evaluation process, to enhance the collection of health statistics, and to enable health service research on disabilities. On September 9 2011, the Health IT Standards Committee’s Clinical Quality Measures Workgroup and the Vocabulary Task Force jointly recommended that, for functional status, International Classification of Functioning, Disability and Health (ICF) be used for coding categories of function, Logical Observation Identifiers Names and Codes (LOINC) be used for assessment instruments, and SNOMED CT be used for appropriate responses [1].
SNOMED CT and LOINC are well-known standard terminologies in biomedical informatics. ICF is a multipurpose classification that, together with ICD, is a reference classification in the WHO Family of International Classifications (WHO-FIC). It provides a standard language and conceptual basis for the definition and measurement of functions and disability. Unlike a medical model of disability, which sees loss of functions only as consequences of diseases and disorders, ICF embodies a “bio-psycho-social synthesis” that conceptualizes function and disability in the context of health conditions, environmental factors, and personal factors. Therefore ICF codes are organized as hierarchies within the ICF components Body Function, Body Structure, Activity and Participation, and Environmental Factors [2].
The Blue Book listing criteria, because of its use in making decisions about claimants’ eligibility, plays a central role in defining the type of data that should captured in the evaluation process. In particular, because several chapters of the Blue Book include criteria that require the assessment of a claimant’s functional capacity, we developed a method to investigate the extent to which ICF and SNOMED CT can be used to code the concepts and relationships appearing in the listings. To derive the concepts and relationship that should be mapped, we decomposed the listing criteria using a structured schema. To take into account inexact matches between listing concepts and terms from ICF and SNOMED CT, we defined different mapping types. We defined various metrics for measuring the coverage of each terminology, taking into account the effects of inexact matches and frequency of occurrence. Finally we performed a proof-of-concept validation of our method by mapping the terms in the disability criteria of Adult Listings, Chapter 12 (Mental Disorders). In this small-sample study, we found that SNOMED CT dominates ICF in almost all the metrics that we have computed.
Methods
The first problem in using ICF to code health and health-related states is that ICF codes (such as “b210 Seeing functions”) denote possible body functions, body structures, activities and participations, and environmental factors that may affect a person’s function. They are not codes for specific conditions that a person may have. These codes need to be combined with qualifiers (such as “3 Moderate Impairment”) in order to describe a health-related state. To map disability criteria into ICF codes, it is necessary to decompose descriptions of a claimant’s physical and mental states into a format that facilitates matching ICF’s specific coding scheme. Much have been written on the representation of eligibility criteria [3]. For this coverage study, we adapted our earlier work on annotating human study eligibility criteria [4] to create a mapping schema that specifies, for a particular listing criterion, how to identify concepts and relationships in the criterion and how to decompose them in a structured format that makes clear the semantic roles played by these concepts and relationships. For example, the criterion
A residual disease process that has resulted in such marginal adjustment that even a minimal increase in mental demands or change in the environment would be predicted to cause the individual to decompensate
can be decomposed into the concept residual disease process having the relationship has result with another complex concept Y, where Y can be structured as a marginal adjustment that is defined by another complex concept a minimal increase in mental demands or change in the environment would be predicted to cause the individual to decompensate. We decompose the concept recursively until we end up with atomic concepts and relationships that cannot be further decomposed. These atomic concepts and relationships are the entries that we want to map to standard terminologies.
More specifically, we define as atomic a Primitive Concept (an entity representing an idea that stands by itself and that we don’t further decompose (e.g., disease process), a Qualifier (a relationship or attribute that can be used to specialize the meaning of a concept; e.g. result in), and a Qualifier value (an attribute value that describes a quality of the thing represented by the concept; e.g., high). Because of our special interest in ICF, we define Function (e.g., short-term memory) as a special kind of Primitive Concept.
We define composite concepts recursively using other concepts and relationships. A Boolean concept is made up of a combination of other concepts using Boolean operators AND, OR, and NOT. A Qualified concept consists of a root concept and a set of Qualifiers and Qualifier values that are implicitly combined using AND. A Change in function concept consists of a concept representing one or a combination of functions and a change descriptor (an instance of Qualifier value), such as impaired or significant loss, that describes how the function has changed. Finally, we have an Event collection concept that represents a set of events (e.g., repeated episodes of decompensation). An instance of Event collection that is associated with a concept (e.g., decompensation) represents multiple occurrences of the event signified by the concept.
We define Mapping as having two properties: a Mapped code and a Mapping type. An instance of Mapped code principally contains attributes that identify the code and terminology of the matched concept. We defined five different types of mappings, as shown in Table 1.
Table 1.
Types of mapping
| Mapping type | Definition |
|---|---|
| equivalence | The mapped code is equivalent to the concept, qualifier value, or qualifier being mapped |
| generalization | The mapped code is a generalization of the concept, qualifier value, or qualifier being mapped |
| specialization | The mapped cod is a specialization of the concept, qualifier value, or qualifier being mapped |
| partial match | The meanings of the mapped code and the concept, qualifier value, or qualifier being mapped overlap |
| related | The mapped code is related to the concept, qualifier value, or qualifier being mapped, although their meanings don’t overlap |
We modeled the mapping schema in Protégé,3 an ontology editor developed at Stanford. In addition to the types of concepts and qualifiers, mapping code, and listing criteria, we have classes that represent the ICF chapters and criteria groups in a chapter. The result is a small ontology consisting of the entities (Figure 2) and properties of the mapping schema.
Figure 2.
Class hierarchy representing entities in the mapping schema. Entities in the Annotatable class hierarchy allow us to represent the Listing_criteria of a Listing_chapter as part of Criteria_groups. The Listing_criteria are decomposed into their constituent Primitive_concept, Qualifiers, and Qualifier_values, which have Mappings that map them to Mapped_code from standard terminologies.
Protégé allows us to model the groupings of the criteria in a Blue Book chapter and to annotate the criteria with concepts, to decompose the concepts, and to map the concepts to codes in terminologies that are available in Bioportal, a repository of biomedical ontologies and terminologies maintained by the National Center for Biomedical Ontologies.4
A Bioportal reference widget [5] for Protégé allows us to search for terminology codes available in Bioportal from within Protégé (Figure 3). We configured this widget for the has_mapped_code property. A user needing a code for the property can search within the widget codes or terms in specified terminologies or ontologies. Once he or she decides that a particular code is appropriate, clicking the “Import” button automatically creates, in Protégé, the mapped code corresponding to the selected code in the search results.
Figure 3.
The use of Bioportal reference widget in Protégé to search for ‘short term memory’ in ICF. The widget matches the search term among all descriptors of entities in the selected ontology and displays the search results (e.g., “b1440. Short-term memory”) as rows in a table. To the right of each search result there are clickable buttons to perform operations such as importing the selected term into Protégé ontology.
Once we had defined the mapping schema, two developers independently decomposed the criteria in the Adult Mental Disorder chapter of the Blue Book and mapped the concepts and relationship derived from the decompositions to ICF and SNOMED CT. They reconciled their work to come up with the consensus decompositions and mappings. To allow people not familiar with Protégé to access the mappings, we wrote scripts that export the mapping into a variety of formats, including HTML and text (Figure 4).
Figure 4.
A display showing the decomposition and mappings of the criterion “Difficulty concentrating or thinking” in a generated HTML file. The decomposition ends up with two instances of Change in function that have {Difficulty with, Concentrating} and {Difficulty with, Thinking} as terms that need mapping. ‘Concentration’ is shown to have equivalence match with ICF code ‘b1400. Sustaining attention’ and ‘Thinking’ is shown to have equivalence match with ICF code ‘d163. Thinking.’
Next we define the metrics for quantifying the mapping results. Evaluations of a terminology’s coverage of a subject domain usually define coverage as the proportion of terms and relationships harvested from a corpus in the subject domain that can be represented using terms and relationships in the terminology [6–12]. The coverage metrics may be refined by assigning “importance” weights to the concepts to be matched [7], by weighing the quality of matches [12], by the use of post-coordinated concepts to match target terms [10], and by the frequency of occurrences of the concept in the corpus [10]. Because the decompositions of different criteria may contain the same concepts, the leaf instances in our decompositions of listing criteria include duplicate concepts. Furthermore, sections of the Mental Disorders chapter may contain criteria that already occur in other sections. Together with the two weighting schemes of the mapping types (equal weights or weights of {1.0, 0.7, 0.7, 0.5, 0.3} for the 5 mapping types {equivalence, generalization, specialization, partial match, related}), we can define 6 metrics as shown in Table 2.
Table 2.
Definition of metrics used to evaluate the coverage of terminologies.
| Metric # | Target terms to be mapped | Weights on mapping types {equivalence, generalization, specialization, partial match, related} |
|---|---|---|
| 1 | Concepts, qualifiers, and qualifier values (with no duplicate) | {1, 1, 1, 1, 1} |
| 2 | Concepts, qualifiers, and qualifier values (with no duplicate) | {1.0, 0.7, 0.7, 0.5, 0.3} |
| 3 | Bag of concepts, qualifiers, and qualifier values that are leaf nodes in the decomposition of distinct Chapter 12 criteria (no repetition of criteria) | {1, 1, 1, 1, 1} |
| 4 | Bag of concepts, qualifiers, and qualifier values that are leaf nodes in the decomposition of distinct Chapter 12 criteria (no repetition of criteria) | {1.0, 0.7, 0.7, 0.5, 0.3} |
| 5 | Bag of concepts, qualifiers, and qualifier values that are leaf nodes in the decomposition of all Chapter 12 criteria (possible repetition of criteria) | {1, 1, 1, 1, 1} |
| 6 | Bag of concepts, qualifiers, and qualifier values that are leaf nodes in the decomposition of all Chapter 12 criteria (possible repetition of criteria) | {1.0, 0.7, 0.7, 0.5, 0.3} |
We wrote code to traverse the consensus Protégé data structure of criteria, their decompositions, and mappings to output the criteria, concepts, and mapping information to a spreadsheet. The computation of metrics was implemented in Excel. We ran the code for SNOMED CT mappings and for ICF mappings.
Because ICF is designed to capture results of functional assessments, it necessarily has poor coverage of nonfunctional concepts. To correct for this bias, we made a tentative assignment of the criteria into functional and nonfunctional groups, and computed the metrics for each group.
Finally, we disaggregated concepts, qualifiers, and qualifier values and examined the coverage of ICF and SNOMED CT for them separately.
Results
In this proof-of-concept experiment, we found that the Adult Mental Disorders chapter of SSA’s Blue Book has 58 distinct “atomic” criteria that are not defined in terms of other criteria. These criteria reference 114 distinct atomic concepts, 18 distinct qualifiers, and 40 distinct qualifier values. There are numerous groupings of criteria. For example, Generalized persistent anxiety is defined in terms of subcriteria such as Motor Tension and Apprehensive expectations. Generalized persistent anxiety itself is part of a group of criteria used to define Anxiety-related disorders. In the tables that display the results of our mappings, “criteria” refers only to the atomic criteria, not the higher-level criteria that are defined in terms of other subcriteria.
Tables 3 and 4 display the coverage metrics of ICF and SNOMED CT for concepts, qualifiers, and qualifier values (Results for ICF mappings have shaded background in the tables for better readability.) As to be expected, SNOMED CT has a higher percentage of mapped concepts. We can also see that functional criteria are often duplicated in different sections of the chapter. So that when we weigh the metric to include all occurrences of concepts anywhere in the chapter (column 4 of each table), the metrics of ICF coverage for the functional criteria are significantly higher than when we consider only mappings of distinct concepts and qualifiers.
Table 3.
Coverage of ICF for concepts, qualifiers, and qualifier values in Mental Disorders criteria. For brevity, “concept” in this table denotes all concepts, qualifiers, and qualifier values.
| Mapping weights equal 1 | |||
| Types of criteria | No concept repetition | No criteria repetition | All occurrences of concept in chapter |
| Functional criteria | 48.48% | 59.69% | 68.63% |
| Non-functional criteria | 37.97% | 40.23% | 31.52% |
| All criteria | 43.82% | 51.85% | 57.09% |
| Mapping weights = {1, 0.7, 0.7, 0.5, 0.3} for {equivalence, generalization, specialization, partial match, related concept} mappings | |||
| Types of criteria | No concept repetition | No criteria repetition | All occurrences of concept in chapter |
| Functional criteria | 38.48% | 44.81% | 50.27% |
| Non-functional criteria | 23.80% | 24.71% | 19.67% |
| All criteria | 31.97% | 36.71% | 40.76% |
Table 4.
Coverage of SNOMED CT for concepts in Mental Disorders criteria. For brevity, “concept” in this table denotes all concepts, qualifiers, and qualifier values.
| Mapping weights equals 1 | |||
| Types of criteria | No concept repetition | No criteria repetition | All occurrences of concept in chapter |
| Functional criteria | 76.47% | 80.18% | 84.53% |
| Non-functional criteria | 80.00% | 79.45% | 83.63% |
| All criteria | 77.37% | 79.89% | 84.25% |
| Mapping weights = {1, 0.7, 0.7, 0.5, 0.3} for {equivalence, generalization, specialization, partial match, related concept} mappings | |||
| Types of criteria | No concept repetition | No criteria repetition | All occurrences of concept in chapter |
| Functional criteria | 66.59% | 71.17% | 73.20% |
| Non-functional criteria | 77.08% | 76.85% | 81.58% |
| All criteria | 70.07% | 73.42% | 75.82% |
What may be surprising in Tables 3 and 4 is that SNOMED CT also dominates when we examine the functional criteria alone. The relatively low coverage rate of ICF, however, can be explained in part by the low coverage of qualifiers and qualifier values, especially qualifiers. If we compare the mapping rates of concepts only, excluding qualifiers and qualifier values, as shown in Tables 5 and 6, the rates of mappings for functional criteria increase substantially. In fact, if we measure in terms of the all occurrence of concepts in functional criteria, ICF has slightly higher coverage (86.7%) than SNOMED CT (84.39%). This finding results from relatively higher frequent occurrences of functional criteria in the Adult Mental Disorders chapter of the SSA’s Blue Book.
Table 5.
Coverage of ICF for concepts (excluding qualifiers and qualifier values) in Mental Disorders criteria.
| Mapping weights equals 1 | |||
| Types of criteria | No concept repetition | No criteria repetition | All occurrences of concept in chapter |
| Functional criteria | 66.67% | 76.92% | 87.60% |
| Non-functional criteria | 46.67% | 56.14% | 47.71% |
| All criteria | 56.99% | 68.15% | 75.49% |
| Mapping weights = {1, 0.7, 0.7, 0.5, 0.3} for {equivalence, generalization, specialization, partial match, related concept} mappings | |||
| Types of criteria | No concept repetition | No criteria repetition | All occurrences of concept in chapter |
| Functional criteria | 52.78% | 58.08% | 64.48% |
| Non-functional criteria | 29.33% | 33.51% | 28.81% |
| All criteria | 41.94% | 47.70% | 53.65% |
Table 6.
Coverage of SNOMED CT for concepts (excluding qualifiers and qualifier values) in Mental Disorders criteria.
| Mapping weights equals 1 | |||
| Types of criteria | No concept repetition | No criteria repetition | All occurrences of concept in chapter |
| Functional criteria | 85.00% | 85.25% | 84.39% |
| Non-functional criteria | 82.61% | 83.67% | 85.71% |
| All criteria | 83.17% | 84.55% | 84.87% |
| Mapping weights = {1, 0.7, 0.7, 0.5, 0.3} for {equivalence, generalization, specialization, partial match, related concept} mappings | |||
| Types of criteria | No concept repetition | No criteria repetition | All occurrences of concept in chapter |
| Functional criteria | 74.50% | 74.92% | 70.12% |
| Non-functional criteria | 78.48% | 79.80% | 82.14% |
| All criteria | 75.35% | 77.09% | 74.46% |
We can also see that the reduction in the metrics when we give inexact matches less weight is smaller in the case of SNOMED CT mappings than that of ICF mappings. This can be explained by the more granular nature of SNOMED CT concepts, which leads to more exact matches.
Discussion
We have developed a method for evaluating the extent to which ICF and SNOMED CT cover the concepts and relationships used to describe SSA Blue Book listing criteria. We applied the method to the criteria of the Adult Mental Disorders chapter and computed a number of metrics for comparing the coverage of the two terminologies. Our findings suggest that, compared to ICF, SNOMED CT has broader coverage of both functional and nonfunctional concepts in a Blue Book chapter that is rich in functional assessment. This result, if validated by larger scale studies not only in the area of listing criteria but also in the vocabulary used in, for example, assessment instruments, raises questions about the roles ICF and SNOMED CT should play in standardizing terminologies used in functional assessment. In an ideal world, ICF and SNOMED CT are harmonized so that SNOMED CT’s more granular vocabulary can enrich and complement ICF’s systematic descriptions of functions, disabilities, and factors that affect them.
The results reported in this paper should be considered preliminary, as the sample size of criteria used in the experiment is small and the concept decomposition, terminology mappings, and characterization of criteria as functional and non-functional had not been checked by experts on disability determination and terminologies. Nevertheless the qualitative results probably hold. SNOMED CT is a much larger terminology than ICF. The range and granularity of SNOMED CT lead to higher coverage rate of concepts used to define Mental Disorders criteria.
The size and complexity of SNOMED CT, however, means that finding the right mappings is a more error-prone process. Furthermore, SNOMED CT allows multiple decomposition and mapping possibilities. In particular, concepts described by SNOMED CT clinical findings often can be defined by a combination of observable entity and qualifier values. For example, the Mental Disorders concept “Impaired memory” can be mapped to either “Impaired memory” (clinical finding 386807006) or a Change function combination of “Impaired” (qualifier value 260379002) and “Memory function” (observable entity 303116000). A more granular mapping (e.g., using the observational entity and qualifier value combination instead of a single clinical finding concept) will drive up the coverage ratio.
The innovation of this work involves the development of a structured decomposition method for identifying appropriate concepts, qualifiers, and qualifier values in complex criteria that should be mapped to standard terminologies. This decomposition defines the semantic roles that the entities play in helping to define the meanings of these concepts. This decomposition is essentially a kind of post-coordination. As such, it can be formalized in description logic. For example, a qualified concept consisting of a root concept C, a qualifier Q, and a qualifier value V is essentially an OWL class expression (using the Manchester Syntax: C AND (Q some V). Given this interpretation of decomposition, we can generate logic expressions that capture much of the meanings of the disability criteria, as was described in [4]. Given appropriate data, we can use a description-logic reasoner to classify the data to determine whether they satisfy the logical expressions, potentially providing a method to help determining whether a claimant is eligible for disability benefits.
Conclusions
We developed a structured decomposition method for identifying appropriate concepts, qualifiers, and qualifier values that occur in eligibility criteria and that should be mapped to standard terminologies. A proof-of-concept application of this method to the Adult Mental Disorders chapter of SSA’s Blue Book listing criteria shows that SNOMED CT has significantly higher coverage of functional and non-functional concepts than ICF. While the method is applicable to terminology coverage studies involving complex criteria, the specific findings of the proof-of-concept experiment need to be further validated by disability determination and terminology experts in larger studies.
Acknowledgments
This work was supported by Interagency Personnel Agreements funded by the Social Security Administration (SSA). We gratefully acknowledge the assistance provided by Derek Wang, Rosemary Hall, Dr. Laurence Desi Sr, Bob Hastings, Joe Herendeen, Ronald Lee, Michele Schaefer, Cheyl Williams, and Jim Twist from SSA and John Hough from the Center for Disease Control and Prevention.
Footnotes
20 C.F.R. §§ 404.1520 and 416.920
More information about Protégé can be found at http://protege.stanford.edu
References
- 1.Health IT Standards Committee: Recommendations to the National Coordinator for Health IT 2011 [2015-03-11] Available from: http://www.healthit.gov/sites/default/files/standards-certification/HITSC_CQMWG_VTF_Transmit_090911.pdf.
- 2.World Health Organization . International Classification of Functioning, Disability, and Health. Geneva, Switzerland: 2001. [Google Scholar]
- 3.Weng C, Tu SW, Sim I, Richesson R. Formal representation of eligibility criteria: A literature review. J Biomed Inform. 2010 Jun;43(3):451–67. doi: 10.1016/j.jbi.2009.12.004. PubMed PMID: 20034594. Pubmed Central PMCID: 2878905. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Tu SW, Peleg M, Carini S, Bobak M, Ross J, Rubin D, et al. A practical method for transforming free-text eligibility criteria into computable criteria. J Biomed Inform. 2011 Apr;44(2):239–50. doi: 10.1016/j.jbi.2010.09.007. PubMed PMID: 20851207. Pubmed Central PMCID: 3129371. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Nair J, Tudorache T, Whetzel T, Noy N, Musen MA. ICBO 2011: CEUR-WS Proceedings. 2011. The BioPortal Import Plugin for Protege. [Google Scholar]
- 6.Ranallo PA, Adam TJ, Nelson KJ, Krueger RF, LaVenture M, Chute CG. Psychological assessment instruments: a coverage analysis using SNOMED CT, LOINC and QS terminology. AMIA Annu Symp Proc. 2013;2013:1333–40. PubMed PMID: 24551411. Pubmed Central PMCID: 3900148. [PMC free article] [PubMed] [Google Scholar]
- 7.Montella D, Brown SH, Elkin PL, Jackson JC, Rosenbloom ST, Wahner-Roedler D, et al. Comparison of SNOMED CT versus Medcin terminology concept coverage for mild Traumatic Brain Injury. AMIA Annu Symp Proc. 2011;2011:969–78. PubMed PMID: 22195156. Pubmed Central PMCID: 3243122. [PMC free article] [PubMed] [Google Scholar]
- 8.Scichilone RA. The benefits of using SNOMED CT and LOINC in assessment instruments. J AHIMA. 2008 Jul;79(7):56–7. PubMed PMID: 18672729. [PubMed] [Google Scholar]
- 9.Elkin PL, Brown SH, Balas A, Temesgen Z, Wahner-Roedler D, Froehling D, et al. Biosurveillance evaluation of SNOMED CT’s terminology (BEST Trial): Coverage of chief complaints. Stud Health Technol Inform. 2008;136:797–802. PubMed PMID: 18487829. [PubMed] [Google Scholar]
- 10.Arts DG, Cornet R, de Jonge E, de Keizer NF. Methods for evaluation of medical terminological systems–A literature review and a case study. Methods Inf Med. 2005;44(5):616–25. PubMed PMID: 16400369. [PubMed] [Google Scholar]
- 11.Brown SH, Elkin PL, Rosenbloom ST, Husser C, Bauer BA, Lincoln MJ, et al. VA National Drug File Reference Terminology: A cross-institutional content coverage study. Stud Health Technol Inform. 2004;107(Pt 1):477–81. PubMed PMID: 15360858. [PubMed] [Google Scholar]
- 12.Chute CG, Cohn SP, Campbell KE, Oliver DE, Campbell JR. The content coverage of clinical classifications. For The Computer-Based Patient Record Institute’s Work Group on Codes & Structures. J Am Med Inform Assoc. 1996 May-Jun;3(3):224–33. doi: 10.1136/jamia.1996.96310636. PubMed PMID: 8723613. Pubmed Central PMCID: 116304. [DOI] [PMC free article] [PubMed] [Google Scholar]