Abstract
Authors studied the United States (U.S.) Department of Veterans Affairs’ (VA) Virtual Lifetime Electronic Record (VLER) Health pilot phase relative to two attributes of data quality – the adoption of eHealth Exchange data standards, and clinical content exchanged. The VLER Health pilot was an early effort in testing implementation of eHealth Exchange standards and technology. Testing included evaluation of exchange data from the VLER Health pilot sites partners: VA, U.S. Department of Defense (DoD), and private sector health care organizations. Domains assessed data quality and interoperability as it relates to: 1) conformance with data standards related to the underlying structure of C32 Summary Documents (C32) produced by eHealth Exchange partners; and 2) the types of C32 clinical content exchanged. This analysis identified several standards non-conformance issues in sample C32 files and informed further discourse on the methods needed to effectively monitor Health Information Exchange (HIE) data content and standards conformance.
Introduction
Through the Virtual Lifetime Electronic Record (VLER) Health Exchange initiative, the United States (U.S.) Department of Veteran Affairs (VA) can electronically share parts of Veterans’ health records with providers at the U.S. Department of Defense (DoD), and participating private sector health care organizations (exchange partners). As a direct benefit to Veterans, doctors involved with their care would have a more comprehensive and timely record of health information, including services received by the Veteran outside of their purview. VLER Health Exchange leverages the policies and technical standards of the eHealth Exchange (formerly the Nationwide Health Information Network or NwHIN) supported by the non-profit, public-private collaborative called Healtheway1, 2. Goals include better informed care providers, improved continuity and timeliness of care, enhanced awareness among all parties, and elimination of gaps in a patient’s health record.
VA engaged in a pilot of VLER Health Exchange to comprehensively test and improve health data standards for effective exchange of Veteran health information across the eHealth Exchange3, 4; to establish models for nationwide health information exchange (HIE); to identify scalable implementation strategies; and to assess early impacts of VLER Health Exchange. VA selected 12 sites with a strong business case for HIE and sought a diversity of characteristics, such as geographic factors (e.g., rural, urban), populations served, the maturity of the HIE organization, and their sustainability models. Four pilot sites participated in a three-way exchange between VA, DoD, and the private sector, and eight sites participated in two-way exchange between VA and the private sector. The pilot period concluded in October 2012. A more in-depth description of the VLER Health initiative and lessons from the pilot phase are reported elsewhere5. After satisfactory completion of the pilot phase, Veterans Health Information Exchange has been expanded nationally, connecting to more and more private sector partners and enabling clinicians at all VA Medical Centers to see externally sourced data.
As the policies and technical systems that serve as the foundation for nationwide HIE continue to mature, and the amount of health data and documents being shared across the eHealth Exchange increases, greater attention is now focused on the quality of data exchanged6. Data quality issues impact the way in which the data reach the intended recipients, are realized in the user interface and, for many user systems, incorporated by the Electronic Health Record (EHR). The exchange of health information between the numerous products and versions of EHR systems requires a standardized method for communicating data that is agreed upon and adopted by entities seeking to share data. During the VLER Health pilot, eHealth Exchange Partners shared data through use of a suite of data transport tools and services called CONNECT. The primary health data standards used to structure the exchanged data included the Health Information Technology Standards Panel (HITSP) C32 and C62 document standards 7.
C32 documents use the Health Level 7 (HL7) Continuity of Care Document (CCD) component to describe the content of a health summary to be created, exchanged, and to summarize a patient’s medical status. The content may include administrative (e.g., demographics, insurance) and clinical information (e.g., problem list, medication list, allergies and test results)8. C32 content standards are comprised of 17 content modules that represent the underlying clinical data in both narrative and structured forms. In addition to C32 structured documents, C62 documents can be used to incorporate unstructured clinical notes and scanned documents (e.g., text file, PDFs, or images such as a scanned image of an electrocardiogram)9.
The technical hurdles of matching patient records, exchanging data from different EHR systems and geographies and then properly rendering the data in a manner that can be effectively used by health care providers are significant. Within the data standards themselves, including the C32, issues of optionality and interpretation create differences in the way the standards are implemented and the way data are mapped across systems. Consequently, even the most diligent HIE development and implementation can produce challenges that hinder effective health data exchange.
The VLER Health pilots and HIE initiatives that preceded VLER Health established an important baseline of understanding as it relates to the adoption and implementation of eHealth Exchange health data standards, and the availability and quality of clinical content of shared Veterans within pilot regions10,11,12.
Methods
Effective sharing of Veteran health information across the eHealth Exchange requires export and import of a validated C32 document with health information. Our study methods established a baseline for two attributes of data quality: 1) compliance of exchange partner’s C32 to current data standards; and 2) the clinical content being provided within VLER Health documents exchanged. Addressed in the first are issues of data quality that occur at the structural level of the C32 exchanged by partners. The second evaluates the types of content, or richness, of information exchanged.
Validation of C32 Data Standards Compliance
All eHealth Exchange partners must comply with the current eHealth Exchange standards. The eHealth Exchange onboarding process during the pilot phase required exchange partners to be able to produce a compliant C32, but did not include formal compliance and content testing. The National Institute of Standards and Technology (NIST) Clinical Document Architecture (CDA) validator was used to validate potential eHealth Exchange partner C32’s for conformance to the HITSP/C32 v2.5 standard. The NIST CDA validator tests the underlying Extensible Mark-up Language (XML) within the C32 to determine whether the schema and data provided conform to the requirements established by the HITSP/C32 v2.5 specification.
For this study, the NIST CDA validation application was downloaded from the NIST website, installed locally and configured with the libraries necessary to check the validity of C32 documents per v2.5 specifications. The HITSP/C32 specification consists of an evolving hierarchy of standards for electronic documentation of health information, including services received by a patient (e.g., Continuity of Care Record (CCR), HL7 CDA, CCD, HITSP C32).
The NIST CDA validator reports the types of non-conformances found in each related section and classifies non-conformance alerts with the current C32 standard into levels of severity that include errors (items of non-conformance), warnings (items that technically conform, but could be better constructed), and notes (general comments and suggestions on implementation). Should a part of the C32 being tested not conform, a report is provided that outlines where and why an error occurred according to the specification (Figure 1).
Figure 1.
Example of NIST CDA Validation Schematron Report of a sample C32 document with errors
These errors in the underlying XML of the C32 when realized on the screen can produce gaps in the information and reduces the quality of the data experienced by the care provider (Figure 2).
Figure 2.
Example of errors in the C32 XML structure when realized in VA VistAWeb EHR
A sample of fourteen populated C32s provided between October 2011 and July 2012 were tested for their conformance to the current C32 standard – one from each of the 12 pilot site private exchange partners’ EHRs and one from the VA and DoD systems. Test results recorded when non-conformance to the specification was found and where (e.g., CDA, CCD), the types of non-conformance identified (e.g., missing data element or attribute), and the sections in which the non-conformances were found (e.g., header, problems/conditions, medications). This assessment only considered those classified by the NIST CDA validator as potential errors in the C32 structure.
Evaluation of C32 Data Content Availability
The clinical content of the C32 was also assessed to better understand the type and amount of clinical data available to and from VA providers during the VLER Health pilot. In July 2012, tests of VLER Health pilot clinical content were conducted by VA as a part of an operations and quality assurance initiative. VA partner C32s were analyzed based on data available between October 2011 and June 2012.
Based on a random sample of 250 Veterans with records available for exchange for each VLER Health pilot site partner, C32s were queried using VistAWeb, and the types of content available were analyzed (e.g., medications, laboratory results, procedures). Eight of the twelve pilot site exchange partners, plus VA, were included in the study, providing a total of 2,250 Veteran records for analyses. Partners that were not included in this sub-study were either not in production or lacked a sufficient number of matched patients at the time the study was conducted.
For each retrieved C32 document, the populated modules were recorded. The study measures consisted of: 1) whether a C32 was returned for each Veteran; 2) for retrieved C32s, whether the Veteran had any eHealth Exchange clinical data available in their populated C32; and 3) if so, the types of clinical modules were populated. Testers only examined whether each C32 clinical module (e.g., medications) contained data, but did not review the C32 for other attributes of the data content such as completeness, display issues or data quality.
Results
VLER Health Exchange Partner C32 Validation
Validation analysis of eHealth Exchange partner C32s indicated that six of the VLER Health exchange partners produced conformant validation results. The other eight C32s tested, however, resulted in some level of error being reported when run through the NIST CDA validator. Two of the C32s tested produced over 10 unique errors of non-conformance to the standard (often the same error is repeated multiple times depending on the content present in the C32). Issues encountered were primarily related to undefined attributes or XML pattern errors, problems found within the document header, and missing data elements or required values as defined by the C32 standard.
It is important to note that the majority of issues identified would not necessarily impact the way in which clinical content was reported. Many issues were due to administrative attributes of the clinical document (e.g., proper inclusion of a country code) that are important to broad health information exchange, but may not be critical to a patient’s treatment. A person viewing this information in an EHR may not even perceive the impact of these types of errors, but as established by the HITSP C32, these specifications are deemed important and mandatory for proper inclusion, and consequently may result in incompatibilities and errors when shared among C32-compliant software.
Outlined in Table 1 are the categories, definitions, and error types for 103 issues identified across eight of the 14 C32s analyzed. As noted previously, depending on how many records are present in the C32, these issues can be reproduced many times, but represent only one main issue in how the C32 is constructed. The frequency of issues is important in terms of the user experience when viewing the document on their screen.
Table 1.
Validation Issues Identified in VLER Health Partner C32 Samples
| Issue Categories | Definitions | Percent of total issues (n=103) |
Prevalence |
|---|---|---|---|
| Non-conformant Attribute or Pattern | An XML attribute or pattern that is unrecognizable by the rules and requirements provided by the C32 schema | 10% | Found at least once in 5 partner C32s, with over 200 different instances found |
| General Header Constraints | Important details regarding the origins and author of the record may be missing or improperly described | 58% | Found at least once in 8 partner C32s, with over 300 different instances found |
| Missing Elements or Required Values | Required data elements and/or coding missing or improperly represented in the C32 | 32% | Found at least once in 8 partner C32s, with over 200 different instances found |
Source: Based on 14 sample C32s, one from each of the VLER Health exchange partners provided between October 2011 and July 2012.
Non-conformance issues were most commonly found in the medication, insurance, laboratory result, allergy, and problem modules. This fact may only be because these were the modules reported most frequently for the C32s tested, or because information for some clinical modules are not sent at all by a particular eHealth Exchange partner at the time of the study.
Types of Clinical Content eHealth Exchange Partners are Capable of Sharing
Of the nine eHealth Exchange partners whose C32s were analyzed for clinical content (eight private partners and VA), the majority of C32s (88 percent) included clinical content in addition to basic demographic data. Table 2 provides a breakdown of clinical content data availability among VLER Health exchange partners by C32 module, the average of C32s that were populated with data for that module, and the range of results across exchange partners.
Table 2.
Data Availability for VLER Health Partners by C32 Module
| C32 Module | Percent of C32s Populated | Range Populated by Partner |
|---|---|---|
| Demographics | 100% | 100% |
| Providers | 86% | 71% ~ 99% |
| Problems | 84% | 52% ~ 99% |
| Allergies | 74% | 13% ~ 99% |
| Encounters | 74% | 35% ~ 94% |
| Medications | 63% | <1% ~ 97% |
| Vital Signs | 53% | 1% ~ 80% |
| Laboratory Results | 46% | 9% ~ 81% |
| Procedures | 42% | 9% ~ 61% |
| Immunizations | 35% | 2% ~ 63% |
Source: Based on a sample of 250 C32s for nine exchange partners (including VA) at the time of the evaluation, plus VA, retrieved July 2012. (n=2,250)
Note: Tests were conducted in July 2012 as a VLER Health quality assurance/operations study. The disclosures are for the period 10/13/2011 – 7/25/2012.
Given the patient matching processes needed to exchange clinical data, basic demographics information was available in 100 percent of the C32s retrieved. This was expected given that demographics information is required for patient matching and exchange of a C32, which were part of the inclusion criteria for this analysis. Provider information was available in 86 percent of the records. Problem list, allergy list, encounters, medications, and vital signs were also available in more than half of the C32s tested. When analyzing the C32s by VLER Health Partners, problem list, allergy list, and medications remain the most commonly available data module; three VLER Health Partners sent these data for more than 75 percent of their Veterans, and another four VLER Health Partners sent these data for at least half of their Veterans.
Subgroup analysis comparing VLER Health Partners shows that problems and allergies were populated the most frequently (seven out of nine partners), followed by medications and laboratory test results (6 out of 9 partners), followed by list of encounters and list of procedures (5 out of 9 partners). Half of VLER Health Partners were able to send at least eight clinical modules, and nearly all VLER Health Partners could send at least five clinical modules, including problem list, allergy list, medication, laboratory test results, and procedures data.
Discussion
The HL7 Consolidated CDA (C-CDA) is the current standard promoted by Meaningful Use 2014, however at the time of the VLER Health pilot the HITSP C32 summary of care record was elected to be the template used to exchange health information through a national-level HIE. The C32 had yet to be implemented and tested in a large-scale, geographically dispersed, production-level exchange. By implementing the C32 standard in a live patient-care environment with multiple exchange partners through the VLER Health program, VA experienced how “optionality” in the specifications can lead to differences of interpretation, uncertainties of implementation, and incompatibilities among compliant software. The NIST CDA validator identified many non-conformance issues in sample C32 files. However, the issues are not reflective of unpracticed adoption of the standards; rather are reflective of the optionality and flexibility in interpretation and implementation built into the standard. These data quality issues related to the implementation of health information standards can have significant implications in effective rendering of the data when sharing data between numerous partners and systems13.
Although the NIST CDA validation system can help ensure compliance with the C32 specification, and therefore greater compatibility with the receiver’s system, it did not yet assist with assessing clinical validity of this information. A small number of C32s were subsequently manually inspected to determine whether they conformed to HITSP specified standard terminologies. Use of clinical coding terminologies (e.g., Systematized Nomenclature of Medicine Clinical Terms (SNOMED CT), Logical Observation Identifiers Names and Codes (LOINC)) helps to ensure semantic interoperability (i.e., meanings of terms and codes translate across systems) and can enhance quality and safety of decision making by clinicians. Results from this informal study suggest that use of clinical coding terminologies was relatively low (less than 30 percent of the terminology coding requirements were met), but further study of the use of data standards and terminologies by provider EHRs would provide more information about this attribute of data quality.
The study of clinical content availability only assessed whether a C32 data domain was present or not and did not assess the data structure, display, or clinical validity aspects that would further inform the usability of the data. As a part of general operations, VLER Health pilot sites compiled clinical content related quality issues (e.g., duplication of data, inconsistencies in formatting and display) based on anecdotes and observations by clinicians and VLER Health staff members when retrieving VLER Health data. The issues identified by VLER Health staff would likely go undetected by the NIST validation system and included problems related to the quality of the data (e.g., little or no clinical data available in the record), inconsistencies in the way the data were presented (e.g., the name of the medication not listed together with the medication allergy, excessive abbreviations, incorrect terminology mapping), and incomplete details that reduced the value of the information (e.g., missing reference ranges for laboratory test results). Further study at this last mile of HIE usability in which the data exchanged is acted upon by the care provider or system who receives the information is critical to understanding future requirements and priorities in health information standards development. VA has expanded its initial data quality assessment activities to include ongoing production validation and scoring of partner CCDs, including the above mentioned data quality aspects.
While significant challenges were encountered in the exchange of VLER Health data during the VLER Health pilot, it is also important to note that eHealth Exchange Partners were reliably providing a core set of valuable clinical content to VA clinicians. Tests confirmed that for sites that are in production and stable, eHealth Exchange data is retrievable for most VA correlated Veterans, and further, that clinical data are available for 91 percent of these correlated Veterans through eHealth Exchange. The VLER Health pilot has supported the VA to better understand the requirements necessary to effectively exchange health information across the eHealth Exchange and to develop a long-term strategy toward better interoperability of content. C32 standard conformance issues, clinical content availability, and the data presentation issues identified during the VLER Health pilot helped inform VA’s work with the Office of the National Coordinator (ONC) Standards and Interoperability Framework in shaping a more robust and interoperable C-CDA standard.
There are two noteworthy efforts to mention that will help construction and testing of more valid and interoperable content among eHealth Exchange partners. The first is the formal requirement that the new eHealth Exchange onboarding body has adopted for content testing. Healtheway guidance explains to implementers what content is required beyond the C32 to reach a meaningful exchange and how this content should be populated and will be tested. For instance, with HITSP, only the Person Information and the Source sections were required, whereas the Healtheway Bridge C32 will require Allergies, Medications, Problems, and Laboratory Results to also be populated14. This is further promoted by Meaningful Use Certification 2014. The second is the opportunity to consider alternative technical ways to implement structured document creation. For example, Model-Driven Health Tools (MDHT) is a project that develops structured document creation and evaluation tools that are programmatically derived and enforced from models of the specifications15.
This study identified the need for improved models of validation and testing of exchange partner data content. Current NIST validation tools include validation guidelines for addressing Meaningful Use requirements for compliant CCDs. As the number of providers exchanging data across the eHealth Exchange increases, “turn-key” tools for validating clinical content for exchange will need to be devised and include strategies for ongoing monitoring and validation of clinical content and standards implementation. Study results have implications related to the complexity of health data interoperability, the evolution of health data exchange standards and its significance to policies such as those embodied in Meaningful Use16. In order to effectively comply with Meaningful Use 2014 Certification requirements and beyond, strict rules on validation are needed, and more consistent methods of testing should be established for providers and HIE organizations17.
Conclusion
The VLER Health pilot served as a live, large-scale, production-level test of eHealth Exchange standards used to exchange clinical data nationally. VA and its partners experienced challenges in validating standards conformance, populating C32 documents, and presenting the data to providers in a consistently accurate, usable fashion. VA continues to work with DoD, Healtheway, ONC, and other stakeholders to leverage the VLER Health pilot experience to create better standards and tools needed by the nation as it moves beyond the barriers of connectedness in health care to the challenges of data integrity, quality, and usability.
References
- 1.Healtheway eHealth Exchange. Healtheway. Mar, 2014. Available at: http://healthewayinc.org/index.php/exchange. Accessed February 2014.
- 2.Bouhaddou O, Cromwell T, Davis M, et al. Translating standards into practice: Experience and lessons learned at the Department of Veterans Affairs. Journal of Biomedical Informatics. 2012;45(4):813–823. doi: 10.1016/j.jbi.2012.01.003. [DOI] [PubMed] [Google Scholar]
- 3.Saef S. The impact of a health information exchange on resource use and Medicare-allowable charges at eleven emergency departments operated by four major hospital systems in a midsized southeastern city: an observational study using clinical estimates. Annals of Internal Medicine. 2013;43(4):S97. [Google Scholar]
- 4.Ross SE, Radcliff TA, Leblanc WG, Dickinson LM, Libby AM, Nease DEJ. Effects of health information exchange adoption on ambulatory testing rates. Journal of the American Medical Informatics Association. 2013;20(6):1137–42. doi: 10.1136/amiajnl-2012-001608. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Byrne CM, Mercincavage LM, Bouhaddou O, et al. The Department of Veterans Affairs’ (VA) implementation of the Virtual Lifetime Electronic Record (VLER): Findings and lessons learned from health information exchange at 12 sites. International Journal of Medical Informatics. 2014 Aug;83(8):537–547. doi: 10.1016/j.ijmedinf.2014.04.005. [DOI] [PubMed] [Google Scholar]
- 6.Kuperman GJ, Blair JS, Franck RA, Deveraj S, Low AF, NHIN Core Services Content Workgroup Developing data content specifications for the nationwide health information network trial implementations. Journal of the American Medical Informatics Association. 2013;17(1):6–12. doi: 10.1197/jamia.M3282. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Bouhaddou O, Bennett J, Teal J, et al. Toward a Virtual Lifetime Electronic Record: The Department of Veterans Affairs Experience with the Nationwide Health Information Network. Paper presented at: AMIA 2012 Annual Symposium; Chicago, IL: 2012. [PMC free article] [PubMed] [Google Scholar]
- 8.Healthcare Information Technology Standards Panel C 32 2009A- HITSP summary documents using HL7 continuity of care document (CCD) component. 2009 [Google Scholar]
- 9.Healthcare Information Technology Standards Panel C62 2009b – unstructured document component. 2009 [Google Scholar]
- 10.Frisse ME, King JK, Rice WB, et al. A regional health information exchange: Architecture and implementation. Paper presented at: AMIA Annual Symposium Proceedings; Washington D.C.: 2008. [PMC free article] [PubMed] [Google Scholar]
- 11.Biondich PG, Grannis SJ. The Indiana network for patient care: an integrated clinical information system informed by over thirty years of experience. Journal of Public Health Management and Practice. 2004 Nov;:S81–6. [PubMed] [Google Scholar]
- 12.Halamka J, Overhage JM, Riccardi L, Rishel W, Shirky C, Diamond C. Exchanging health information: local distribution, national coordination. Health Affairs. 2005;24(5):1170–1179. doi: 10.1377/hlthaff.24.5.1170. [DOI] [PubMed] [Google Scholar]
- 13.Williams C, Mostashari F, Mertz K, Hogin E, Atwal P. From the Office of the National Coordinator: The strategy for advancing the exchange of health information. Health Affairs. 2012 Mar;31(3):527–36. doi: 10.1377/hlthaff.2011.1314. [DOI] [PubMed] [Google Scholar]
- 14.Healtheway Official eHealth Exchange Technical Errata and Change Log. Healtheway. Mar, 2014. Available at: http://healthewayinc.org/index.php/resources/exchange-specifications?highlight=WyJicmlkZ2UiLCJjMzIiLCJicmlkZ2UgYzMyIl0. Accessed March 2014.
- 15.Model Driven Health Tools MDHT Project Home. Open Health Tools. 2013. Available at: https://www.projects.openhealthtools.org/sf/projects/mdht/
- 16.Marcotte L, Seidman J, Trudel K, et al. Achieving meaningful use of health information technology: A guide for physicians to the EHR incentive programs. Archives of Internal Medicine. 2012;172(6):731–736. doi: 10.1001/archinternmed.2012.872. [DOI] [PubMed] [Google Scholar]
- 17.D’Amore JD, Mandel JC, Kreda DA, et al. Are Meaningful Use Stage 2 certified EHRs ready for interoperability? Findings from the SMART C-CDA Collaborative. Journal of the American Medical Informatics Association. 2014 doi: 10.1136/amiajnl-2014-002883. Epub ahead of print. [DOI] [PMC free article] [PubMed] [Google Scholar]