Evaluation of Discrepancies Among National Library of Medicine (NLM) Value Set Authority Center (VSAC) ICD-10-CM Value Sets: Case Study for Diagnoses of Common Chronic Conditions, Implications, and Potential Solutions

Abstract

The National Library of Medicine (NLM)’s Value Set Authority Center (VSAC) is a crowd-sourced repository with a potential for substantial discrepancy among value sets for the same clinical concepts. To characterize this potential problem, we identified the most common chronic conditions affecting US adults and assessed for discrepancy among VSAC ICD-10-CM value sets for these conditions. An analysis of 32 value sets for 12 conditions identified that a median of 45% of codes for a given condition were potentially problematic (included in at least one, but not all, theoretically equivalent value sets). These problematic codes were used to document clinical care for potentially over 20 million patients in a data warehouse of approximately 150 million US adults. Users of VSAC diagnosis value sets should be cognizant of the prevalence of these discrepancies and take proactive steps to mitigate their impact. Further research is warranted to characterize and address this issue.

Introduction

The Value Set Authority Center (VSAC) is a public data repository for value sets created by different programs, agencies, organizations, and research groups.¹ The VSAC is hosted by the National Library of Medicine (NLM) together with the Office of the National Coordinator for Health Information Technology (ONC) and the Centers for Medicare & Medicaid Services (CMS). Value sets are lists of standardized codes and corresponding terms, which define different clinical concepts (e.g., abdominal x-ray, Coronavirus Disease 19 [COVID-19], angiotensin converting enzyme [ACE] inhibitors) and are based on standard clinical terminologies such as the Logical Observation Identifiers Names and Codes (LOINC), the Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT), RxNorm, and the International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM).

Accurate value sets are critical for identifying clinical concepts and achieving semantic interoperability.² For example, take the case of when a clinical decision support (CDS) system evaluates whether a patient should be prescribed an ACE inhibitor due to the presence of hypertension and chronic kidney disease, the lack of a known adverse reaction to ACE inhibitors, and the lack of an active current prescription for an ACE inhibitor. In order for this CDS system to make an accurate assessment and recommendation, accurate value sets are needed to characterize the patient’s status with regard to diseases, medications, and adverse reactions based on the data in their electronic health record (EHR).³ If, for example, the hypertension value set is missing an ICD-10-CM code, and that code is what is used in that patient’s care to identify the patient as having this condition, this incomplete value set could result in the patient not being offered this evidence-based care guidance. Beyond CDS, value sets serve as a foundational resource for various important clinical applications including electronic clinical quality measurement, clinical data registries, public health reporting, and health information exchange.⁴

Currently, the VSAC serves as the primary publicly accessible repository of value sets in the United States (US). For example, the Council of Medical Specialty Societies advocates using the VSAC as a source for distributing and standardizing new clinical concepts.⁵ As another example, the Agency for Healthcare Research and Quality (AHRQ)’s CDS Connect Authoring Tool uses the VSAC for identifying clinical concepts in its clinical decision rules.⁶ There are three types of value sets in the VSAC: extensional value sets, in which all codes are specifically enumerated; intensional value sets, where codes are defined in terms of logical statements and concepts are dynamically generated; and grouping value sets, which represent a union of extensional or intensional value sets.⁷

The NLM does not centrally curate VSAC value sets.⁴ Instead, the NLM enables external organizations and even individual contributors to act as value set stewards that develop and maintain the value sets contained in the VSAC.⁸ VSAC stewards include the National Committee for Quality Assurance, which defines the widely used Healthcare Effectiveness Data and Information Set (HEDIS) clinical performance measures⁹; the Joint Commission, an organization that evaluates and accredits healthcare organizations; and the Centers for Medicare & Medicaid Services (CMS), a federal agency that provides healthcare coverage through Medicare and Medicaid and defines a variety of electronic clinical quality measures (eCQMs).¹⁰

Given the critical importance of VSAC value sets in a variety of clinical applications, there have been several evaluations of the accuracy and completeness of these value sets in the literature. In 2013, Winnenburg and Bodenreider compared VSAC value sets for eCQMs against value sets they generated by identifying all descendants of root concepts in those value sets.¹¹ In this analysis, they identified that a majority of the evaluated value sets may have missed one or more relevant codes.¹¹ Then, in 2017, Cholan et al. reported that VSAC value sets for the same eCQM developed by two different organizations had multiple discrepancies.¹² Furthermore, in 2019, Chu et al. reported that ten commercial intensional SNOMED CT value sets licensed at their health system for clinical conditions were substantially more complete than corresponding extensional value sets obtained from the VSAC.¹³

While these prior studies clearly identified potential problems in the accuracy or completeness of VSAC value sets, most prior studies of VSAC value sets did not include an evaluation of the actual impact of potentially relevant codes missing from a value set. For example, even if a given value set contains only 65% of the codes as another value set, the practical difference could be minimal if the remaining 35% of codes were not used, or used minimally, in the actual care of patients. Cholan et al. did conduct an impact analysis of the discrepancies in VSAC value sets for eCQM assessment, but this analysis was done on a clinical data set comprising only five clinics.¹²

Given the informatics community’s reliance on VSAC value sets, including by our own research team for a variety of CDS use cases, the objective of this study was to contribute to the literature through a multi-faceted evaluation of the accuracy and completeness of VSAC value sets that included an impact analysis using a large, representative patient data set. Using twelve common chronic conditions as the focus of analysis, this study evaluated (i) the degree to which VSAC value sets for each condition had discrepancies indicative of potential issues; (ii) the degree to which these VSAC value sets were up-to-date with regard to codes introduced in the past 5 years; and (iii) the frequency with which problematic codes from the first two analyses were used in the actual care of over 150 million patients.

According to the National Center for Chronic Disease Prevention and Health Promotion, six out of ten adults in the US have a chronic disease, with chronic conditions representing the leading cause of death, disability, and health care expenditures.¹⁴ Given the prevalence and importance of chronic diseases, the analyses of VSAC value sets in this study were conducted the context of common chronic conditions.

Methods

Overview. This study consisted of three primary analyses of VSAC value sets. An overview of these analyses is provided here, followed by a detailed description of these methods.

The first analysis was an examination of inter-value set reliability. This was done by identifying 12 common chronic conditions and determining the degree to which there were apparent inconsistencies between the VSAC value sets available for these conditions. Only value sets with names and metadata indicating that they should have identical semantics were included in the analysis. The rationale for this analysis was that discrepancies in such value sets represent potential problems, either due to the inclusion of a code which should not have been included, or the omission of a code which should have been included. This analysis also established a superset of VSAC codes for each condition.

The second analysis was an examination of value set up-to-dateness. This was done by taking the superset of VSAC codes for each chronic condition and determining whether new codes that had been introduced in the past 5 years were included in the individual value sets or the VSAC superset for each condition. The rationale for this analysis was that even a VSAC value set that was accurate and complete at the time of initial definition may become outdated. The third analysis was an impact analysis. For this analysis, a large data warehouse (Epic Cosmos) was used to evaluate the frequency with which potentially problematic codes identified in the first two analyses were actually used in clinical care. For this study, “potentially problematic codes” consisted of (i) codes that were included in at least one, but not all, VSAC value sets for a condition, as well as (ii) codes introduced in the past 5 years that were missing from a VSAC superset for a condition.

All analyses were conducted using ICD-10-CM. This was done because ICD-10-CM was commonly used for defining VSAC value sets for conditions, and as ICD-10-CM is supported in Epic Cosmos for diagnoses.

Two physicians were involved in making clinical judgments required for the study. One physician (NL) served as the primary reviewer for all analyses. Where there was uncertainty by the primary reviewer, a second physician (KK) served as a secondary reviewer, and the final decision was made through discussion and consensus.

Data Sources. Two data sources were used for this study. The first data source was the VSAC.¹ The second data source used in this study was Epic Cosmos.¹⁵ Cosmos is a HIPAA limited data set of patients from health systems that use Epic software. As of March 2023 when this work was conducted, the Cosmos data set included more than 183 million patients from over 190 health systems. The Patient data model in Cosmos was used to assess the number of patients for whom a particular diagnosis code had been used for an encounter diagnosis, admitting diagnosis, billed final diagnosis, billed admitting diagnosis, billed charge-associated diagnosis, or an active diagnosis on the problem list. We restricted our search to patients aged 18 years or older, and we checked patient records from January 1, 1999 until March 1, 2023. After applying this restriction, 150,466,864 patient records were analyzed for this study.

Selection of Target Chronic Conditions and Associated Value Sets. For selecting chronic conditions to analyze, we first used CMS’s disease prevalence data from 2007-2018¹⁶ to identify 24 of the most common chronic conditions affecting US adults. The chronic conditions screened for this study were alcohol abuse, drug abuse/substance abuse, Alzheimer’s disease and related dementia, heart failure, arthritis (osteoarthritis and rheumatoid arthritis), hepatitis (chronic viral B and C), asthma, HIV/AIDS, atrial fibrillation, hyperlipidemia (high cholesterol), autism spectrum disorders, hypertension (high blood pressure), cancer (breast, colorectal, lung, and prostate), ischemic heart disease, chronic kidney disease, osteoporosis, chronic obstructive pulmonary disease (COPD), schizophrenia and other psychotic disorders, depression, stroke, and diabetes.¹⁶

The VSAC “Search Value Sets” tool¹⁷ was used to search for relevant value sets and associated metadata for each chronic condition. ICD-10-CM extensional value sets, ICD-10-CM intensional value sets, and grouping value sets were retrieved.

For each condition, a physician reviewed the names, descriptions, and inclusion and exclusion criteria for each potentially relevant value set and included those value sets which matched the targeted condition. In order to reduce the chances of spurious discrepancies being found across similar, but intentionally different, value sets related to a condition, a conscious effort was made to err on the side of caution and to only include value sets for which the name and metadata clearly referenced the same condition. For example, for heart failure, value sets with the name “congestive heart failure” rather than “heart failure” were excluded, as were value sets for which the metadata indicated a focus on a specific subtype of heart failure such as congestive heart failure.

Duplicate value sets were consolidated. For example, if a given steward defined both an extensional ICD-10-CM value set for a condition and a grouping value set for the same condition containing both this extensional value set for ICD-10-CM and another extensional value set for SNOMED CT, the ICD-10-CM value set was only included once in the analysis. A condition was included for further analysis if there were two or more VSAC value sets available from different stewards. The final analysis was conducted on data downloaded from the VSAC on March 2, 2023.

Inter-Value Set Reliability Analysis. The reliability of value sets within a given condition was analyzed through the following six steps for each included chronic condition.

• Step 1: Filtering for ICD-10-CM in grouping value sets. In this first step, grouping value sets were processed and filtered for ICD-10-CM value sets.

• Step 2: Superset generation. In this step, we formed a superset of all unique codes by identifying all unique concepts included in any of the individual value sets.

• Step 3: Identification of potentially problematic codes. Potentially problematic codes were identified by searching for codes that were in the superset but missing from at least one constituent value set.

• Step 4: Impact analysis using Epic Cosmos. For each potentially problematic code, the number of patients in Epic Cosmos that had the code used in their actual clinical care was assessed as a measure of the potential impact of the issue.

• Step 5: Descriptive statistics. For each chronic condition, as well as for the data set as a whole, descriptive statistics were computed.

Value Set Up-To-Dateness Analysis. The up-to-dateness of value sets within each condition was analyzed through the following three steps.

• Step 1: Identification of recently added ICD-10-CM codes. We identified all ICD-10-CM codes that were added to the code system in the past 5 years by using the change log section of an ICD-10-CM Web site.¹⁸

• Step 2: Determination of whether recently added codes were missing from VSAC supersets. Each of the recently added ICD-10-CM codes were manually reviewed to evaluate whether they were relevant to a study condition. If so, the superset for each VSAC condition was reviewed to identify whether a relevant recent code was missing.

• Step 3: Impact analysis using Epic Cosmos. Similar to the inter-value set reliability analysis, recently added codes that appeared to be inappropriately missing from the superset of a condition were assessed in Epic Cosmos with regard to their frequency of clinical use.

All analyses were conducted using Python 3.11. The pandas package was used to programmatically process the Excel files downloaded from the VSAC, and the matplotlib package was used to generate graphs. For Epic Cosmos analyses, the number of affected patients were assessed individually for each code, then added to obtain an indication of the potential magnitude of the impact. This provided the theoretical maximum number of patients affected by the problematic codes.

Results

Chronic Conditions and Value Sets Included for Analysis. Figure 1 summarizes the selection of chronic conditions and associated value sets. For the 24 most prevalent chronic conditions included in the initial search, 1,845 VSAC value sets were identified for screening. Physician filtering of these value sets based on name resulted in 89 value sets remaining for further screening. Inspection of the metadata for the remaining value sets resulted in 44 value sets that had at least 2 value sets that were determined to be semantically equivalent for a given condition. The last step involved removing grouping value sets when a corresponding extensional value set already existed; if a condition no longer had at least 2 semantically equivalent value sets for the condition at this point, that condition and value set were removed. The final set included 32 value sets for 12 conditions. The value sets and conditions included in the analysis are listed in Table 1. As noted in the table, 31 of 32 value sets were stewarded by organizations rather than individual contributors. All 32 included value sets were extensional in nature.

Figure 1.

Selection process flowchart

Table 1.

Chronic conditions and VSAC value sets analyzed

Value Set Name	Value Set Object Identifier (OID)	Value Set Steward
Atrial Fibrillation
Atrial fibrillation	2.16.840.1.113762.1.4.1200.208	Cliniwiz
Atrial Fibrillation	2.16.840.1.113883.17.4077.2.1004	American College of Emergency Physicians/AMA-PCPI
Asthma
Asthma	1.22.3	Cliniwiz
Asthma	2.16.840.1.113762.1.4.1106.60	American College of Emergency Physicians/AMA-PCPI
Asthma (Disorders) (ICD10CM)	2.16.840.1.113762.1.4.1146.1390	Council of State and Territorial Epidemiologists Steward
Asthma	2.16.840.1.113883.3.526.2.60	American Heart Association, Inc.
Breast cancer
Breast Cancer	2.16.840.1.113762.1.4.1047.362	Oncology Nursing Society
Breast Cancer	2.16.840.1.113883.3.1434.1000.1095	College of American Pathologists Steward
Breast Cancer	2.16.840.1.113883.3.526.2.97	PCPI Foundation
Chronic Hepatitis
Chronic Hepatitis	2.16.840.1.113762.1.4.1078.115	Optum
Chronic Hepatitis	2.16.840.1.113762.1.4.1200.212	Cliniwiz
Chronic Obstructive Pulmonary Disease (COPD)
Chronic Obstructive Pulmonary Disease (COPD)	2.16.840.1.113762.1.4.1138.737	Change Healthcare
COPD	2.16.840.1.113762.1.4.1200.228	Cliniwiz
Chronic obstructive pulmonary disease (COPD) Diagnosis	2.16.840.1.113762.1.4.1222.1466	HL7 Patient Care Work Group
COPD	2.16.840.1.113883.3.666.5.776	Lantana
Depression
Depression	2.16.840.1.113762.1.4.1182.279	Change Healthcare
Depression	2.16.840.1.113762.1.4.1248.169	American Institutes for Research
Diabetes
Diabetes	2.16.840.1.113762.1.4.1138.739	Change Healthcare
Diabetes	2.16.840.1.113883.3.464.1003.103.11.1002	National Committee for Quality Assurance
Heart Failure
Heart Failure	1.7.1	Cliniwiz
Heart Failure	2.16.840.1.113762.1.4.1106.82	American College of Emergency Physicians/AMA-PCPI
Hypertension
Hypertension	2.16.840.1.113762.1.4.1032.9	MITRE
Hypertension, Primary and Secondary Diagnosis	2.16.840.1.113762.1.4.1222.1547	HL7 Patient Care Work Group
Hypertension	2.16.840.1.113883.3.3157.4021	Lewin EH Steward
Hypertension	2.16.840.1.113883.3.464.1003.104.11.1038	National Committee for Quality Assurance
Osteoporosis
Osteoporosis (ICD10CM)	2.16.840.1.113762.1.4.1034.631	American Academy of Neurology
Osteoporosis	2.16.840.1.113762.1.4.1200.147	Cliniwiz
Prostate Cancer
Prostate Cancer	2.16.840.1.113762.1.4.1116.306	American Society of Clinical Oncology
Prostate Cancer	2.16.840.1.113883.3.464.1003.108.11.1138	National Committee for Quality Assurance
Prostate Cancer	2.16.840.1.113883.3.526.2.91	Mathematica
Schizophrenia
Schizophrenia or Psychotic Disorder	2.16.840.1.113883.3.464.1003.105.11.1162	MN Community Measurement
Schizophrenia	2.16.840.1.113883.3.464.1003.105.11.1210	National Committee for Quality Assurance

Inter-value set Reliability Analysis. Only 1 of the 12 chronic conditions analyzed had fully matching value sets (heart failure). In the remaining 11 conditions, a total of 189 potentially problematic codes were identified (Figure 2a). The percentage of potentially problematic codes per condition ranged from 0.0% to 88.6%, with an interquartile range of 35.0%, a median value of 45.0%, and a mean value of 42.3%.

Figure 2.

Inter-value set reliability and impact analysis by condition. Figure 2a on the left depicts the number of common codes (i.e., codes included in every value set for the condition) as well as the number of potentially problematic codes (i.e., the codes included in the superset for the condition but not all component value sets). Figure 2b on the right depicts the theoretical maximum number of patients affected by potentially problematic codes for each condition. This was calculated as the sum of the number patients in Epic Cosmos for whom each potentially problematic code was used.

The actual use of these potentially problematic codes in Epic Cosmos is depicted in Figure 2b. Nine of the codes were missing from Epic Cosmos, so the analysis was restricted to the remaining 180 codes. The theoretical maximum number of patients who could be impacted by these potentially problematic codes was 27,522,557 patients (the sum of all the counts in Figure 2b). The largest contributor to this number was code F32.A (depression, unspecified). Some of the least important codes in terms of the number of patients potentially impacted, but with at least one actual use, were J45.9 (other and unspecified asthma), I13.1 (hypertensive heart and chronic kidney disease without heart failure), and M80.8AXK (other osteoporosis with current pathological fracture, other site, subsequent encounter for fracture with nonunion).

Value set Up-To-Dateness Analysis. A total of 3,072 ICD-10-CM codes were identified as having been added in the past 5 years. Each of these codes were clinically reviewed to determine their relevance to the diseases in question. This examination found that only 1 relevant code was missing from the condition supersets. This code was Z79.85 (Long-term [current] use of injectable non-insulin antidiabetic drugs). We were not able to assess the utilization of this code in Epic Cosmos as it was not available for selection.

Discussion

Value sets are an essential foundation for many critical biomedical informatics applications including clinical quality measurement, clinical decision support, and health information exchange. The NLM’s VSAC is an important public resource for value sets, but relatively limited research has been published on the quality of the value sets included in this repository. In this study, we built on prior research evaluating the VSAC by systematically assessing 32 VSAC value sets for 12 of the most common chronic conditions. This analysis identified that there were substantial discrepancies between the value sets. Only 1 of the 12 conditions had fully matching value sets, and on average, 42.3% of the codes identified with a given condition were potentially problematic (i.e., were included in some but not all value sets for the condition). Moreover, the frequency of actual clinical use of these potentially problematic codes was substantial in the Epic Cosmos database. Among approximately 150 million adult patients in this data set, the theoretical maximum impact of these codes was for 27.5 million patients. While the actual impact is most certainly lower, as the patients may have been appropriately identified as having the condition from non-problematic codes, or as the same patient may have been counted multiple times through different problematic codes, we believe our findings quite clearly point to the fact that there is a real potential for issues with VSAC value sets leading to inaccurate characterizations of patient conditions. On the other hand, our analysis of value set up-to-dateness indicated that the superset of VSAC codes for each condition appeared to be consistently capturing most of the relevant codes recently added to ICD-10-CM.

Strengths and Limitations. An important strength of this study is that it conducted an empirical assessment of the potential impact of identified issues by evaluating the frequency with which potentially problematic codes were used in the clinical care of over 150 million adult patients. A second strength is that 12 of the most common chronic conditions affecting US adults were evaluated. Given the prevalence and public health impact of these conditions, identifying potential issues in a foundational informatics resource used for the identification and management of these conditions is important. As a third strength, this study used multiple methods for assessing the quality of VSAC value sets, including a combination of discrepancy analysis, up-to-dateness analysis, and impact analysis.

One limitation of the study is that its scope, while covering many of the most common chronic conditions, is still limited compared to the full scope of value sets included in the VSAC. Specifically, we only examined a subset of conditions and assessed only ICD-10-CM value sets. Therefore, our findings may not necessarily translate to rarer conditions, value sets consisting of other code systems, or value sets for other types of clinical concepts such as medications. Another limitation is that we did not independently establish a gold standard value set for each condition. However, even without a gold standard, when value set stewards disagree on the inclusion of a particular code, such a discrepancy can only be explained by an “error” in one of the value sets. As a final limitation, commercial value sets were not included in the analysis. However, the VSAC is most useful for stakeholders who do not subscribe to commercially provided value sets, so we believe an analysis focused on publicly available VSAC value sets is appropriate.

Implications and Future Directions. This study has several implications. First, this study adds to the literature identifying that VSAC value sets can have meaningful quality issues. Thus, while the VSAC is an invaluable resource for ourselves and many other groups, stakeholders using these value sets must do so with caution. Moreover, further research is warranted on assessing and improving the quality of VSAC value sets. If possible, we feel additional public investment in establishing and curating VSAC value sets would be highly beneficial and in the public good. There is already precedence for this type of activity in terms of how the NLM manages and curates RxNorm as a gold standard medication terminology resource in the US. Another implication of this study is that stakeholders using the VSAC may benefit from identifying discrepancies that exist between value sets for the same clinical concept and explicitly evaluating whether those codes should be included or excluded for their use. We believe tooling support for such a review would be valuable, in particular if coupled with the ability to evaluate the actual clinical utilization of potentially problematic codes so that attention can be focused on those codes that are more frequently used. Finally, as was called upon a decade ago by Winnenburg and Bodenreider¹¹, we recommend that value set stewards contributing to the VSAC work together to define consensus value sets for clinical concepts of mutual interest. Coordinating such harmonization work may be a promising target for high-impact investment by agencies such as the NLM, CMS, or ONC.

Conclusions

In this study, an evaluation of 32 VSAC value sets for 12 common chronic conditions identified substantial discrepancies between value sets for the same conditions. Moreover, many of these discrepant codes were used in actual clinical care when evaluated in the Epic Cosmos database. Given the importance of VSAC value sets in clinical quality improvement activities including eCQM and CDS, further research is needed to assess the quality of VSAC value sets and to identify approaches to improving this valuable public resource.