A refined methodology for validation of information models derived from flowsheet data and applied to a genitourinary case

Abstract

Use of electronic health record data is expanding to support quality improvement and research; however, this requires standardization of the data and validation within and across organizations. Information models (IMs) are created to standardize data elements into a logical organization that includes data elements, definitions, data types, values, and relationships. To be generalizable, these models need to be validated across organizations. The purpose of this case report is to describe a refined methodology for validation of flowsheet IMs and apply the revised process to a genitourinary IM created in one organization. The refined IM process, adding evidence and input from experts, produced a clinically relevant and evidence-based model of genitourinary care. The refined IM process provides a foundation for optimizing electronic health records with comparable nurse sensitive data that can add to common data models for continuity of care and ongoing use for quality improvement and research.

INTRODUCTION

Information models (IMs) are an important means to standardize in electronic health records (EHRs). In the EHR, flowsheets are used by nurses for the periodic and repeated documentation of patient specific data (eg, clinical observations, response to treatment). However, healthcare organizations configure EHR flowsheets in diverse formats, limiting data sharing.^1,2 Use of flowsheet data IMs can aid the standardization of clinical concepts at an atomic level of granularity, define concept relationships, semantic rules, and identify data elements for extraction and retrieval for comparison across settings.³ As a result, flowsheet-based IMs can enhance EHRs to improve consistency of practice documentation, contribute to prioritizing process improvements, and enhance collaboration across organizations to support research.⁴ In 2017, the research team developed a process to validate a Pain IM derived from flowsheet data.⁵ Based on the Pain IM development experience, researchers from 10 organizations proposed using the Pain IM validation process for subsequent clinical topics, except the process needed refinement.

The refinement of the IM process enhances and enables flowsheet IMs to be evidence-based and can contribute to other common data models (CDMs) to support research. For example, the Observational Health Data Sciences and Informatics (OHDSI) network. The OHDSI leverages the Observational Medical Outcomes Partnership (OMOP) CDM (https://www.ohdsi.org/data-standardization/the-common-data-model/) to develop reliable evidence from more than 2500 distinct users across 6 continents from over 100 different databases. The Patient-Centered Outcomes Research Institute (PCORI) is another CDM (https://pcornet.org/data-driven-common-model/) that supports research across organizations to demonstrate the effectiveness of treatment options. There is a gap, however, as neither the PCORI CDM nor OHDSI OMOP contains the care delivery data available in EHR flowsheets. Flowsheet IMs are evolving,^4,6–10 but validation of these across organizations is limited today.^2,5

This case report describes the refinement of the IM flowsheet validation process and the application of the revised process to validate a genitourinary (GU) IM developed from one organization across multiple organizations. This process subsequently can be used to validate additional flowsheet IMs in the future, which can further enhance CDMs to support research.

EVOLUTION OF FLOWSHEET IM VALIDATION PROCESS

The initial process of validating IMs from flowsheet data began in 2017. The research team is a self-selected group of nurse informaticians attending the Nursing Knowledge Big Data Science (NKBDS) Conference at the University of Minnesota School of Nursing.¹¹ The team consisted of 10 organizations (8 contributing data). The purpose was to identify an optimal set of data elements using an IM validation process across organizations. The IM process consisted of extracting de-identified metadata from each participating organization representing flowsheet-based pain concepts in current use and importing them into a secure server. Data mapping software was used to aggregate data and support the team to map their data to assessment, intervention, goal, and outcome concepts and value sets (choice lists) with definitions created. Mappings were compared and discussed biweekly. Consensus was guided by agreement that concepts with usage >50% were retained, 30%-50% discussed, and < 30% removed. A concept comparison with Logical Observation Identifiers Names and Codes (LOINC) was conducted with some concepts renamed for term consistency with LOINC. Research members also informally consulted subject matter experts (SMEs) within their organizations. The validated Pain IM had 30 concepts, 4 panels, and 396 value set items with descriptions and definitions.

Revised IM validation process

The revised IM methodology built on the validation of the Pain IM and was supplemented by an updated review and synthesis of the literature. A medical librarian developed a strategy to search PubMed, OVID and Scopus using the terms: (Ontolog* or data model*) and design* or develop*; restricted to English language from 2000 to 2018, resulting in 226 articles. Titles and abstracts were reviewed with 19 articles selected and analyzed by categories: aims/questions, theory, definitions, clinical topic/domain, method, setting, data source for analysis, findings, and strengths and limitations. Major additions for consideration were the inclusion of evidence-based practice guidelines, data standards, and a survey of SMEs. The team verified practice trends and refined the IM validation process to develop a “future-based” IM, rather than a prior or current practice IM.

Figure 1 summarizes a comparison of the original and revised IM validation processes. Terms identified in blue italics indicate the revised process.

Figure 1.

Comparison of the original and revised methods to validate information models. Blue/italicized text indicates a revised process. EHR: electronic health record.

Some steps were implicit in the original validation process and made explicit in the revision. The incorporation of the literature search to support the revised IM validation process is described next in the scope, analysis, design, validate, and disseminate stages. Application of the revised process to validation of the GU model concludes each section.

Scope

The scope of an IM validation process includes the composition of the modeling team, clinical topic, the purpose, and completeness. The composition of the team can vary, and can include clinical, informatics, and technical members.^12–15 If a particular clinical topic is addressed, the inclusion of SMEs is recommended.^12–15 Their participation can range from the development stage through the evaluation stage.^12–15 In some cases, SMEs are part of the team during the entire process; in other cases, SMEs are consulted after model development to validate results. When SMEs participate in the full project, there is a risk of their ongoing availability over a long period with diminishing input. One strategy is to invite new SMEs to participate while the established SMEs engages in other IM project aspects such as specifying data elements and classes within a topic domain.

Clinical topics, purpose, and completeness of IMs vary. Examples of clinical topics are physiological nursing assessment data,¹⁵ pain,^5,9,16 skin and wound assessments,²hospital-acquired pressure ulcers,¹³ obesity management,^17,18 nursing assessment of cancer survivors,¹⁹ medical errors,²⁰ antibiotic decision support,²¹ and drug surveillance.²² The purpose of IMs can be descriptive, prescriptive, or comparative. If a model is descriptive, the focus is on the current status. If prescriptive, the model focus is on improvement. A comparative model supports benchmarking between and among organizations.²³ The completeness of an IM can be either a minimal or optimal dataset^2,13,19 or a fully detailed clinical model.³

Application

The refined IM scope was applied to GU documentation from 10 mid- to large-sized health care settings and included (1) assuring broad expertise of the research team, (2) specifying a clinical topic, (3) defining the overall purpose of an IM, and (4) identifying the desired completeness. Expertise included proficiency in analysis, EHR design, data extraction, and familiarity with standardized terminologies. The team has extensive experience with EHR vendors and knowledge of each EHR’s documentation strengths and limitations. A reference GU IM created by one participating organization⁷ was selected for IM validation based on the documentation importance of GU data for the delivery of nursing care and nurse sensitive quality measures such as catheter-associated urinary tract infection and urinary incontinence (UI). The overall purpose of IM validation was to create a comparative model, standardizing an essential core set of data elements for assessments, interventions, and outcome concepts, along with definitions and the value sets for inpatient care. The team chose to define an optimal set (completeness) for the direct care nurse or clinician in a hospital, emergency department, or rehabilitation setting. Exclusions were specialized assessments required in postacute and long-term care settings or concepts used primarily by advanced practice nurses or other clinicians.

Analysis

Analysis for validation of an IM is the evaluation of multiple resources for inclusion or exclusion of concepts, definitions, and value sets. Multiple resources guide the selection of concepts for inclusion of content in IMs. These include practice guidelines, validated instruments and scales, and published standards.^3,5,22,24 Use^3,16 and examination of data from practice (ie, flowsheets, template notes, or forms) helps to identify current practice.^13,15,24,25 Informatics standards, such as clinical element models,^3,12,22,26 existing taxonomies,²⁰ or ontological repositories, provide additional IM content.^17,21

Application

The optimal set of concepts, definitions, and value sets was based on deriving key IM data from the EHRs in practice from 8 of the 10 participating organizations, a review of the literature, comparison with current data standards, and numerous iterative discussions of concept uses in practice (informal use cases). The team met biweekly for 1.5 hours over 23 months to achieve consensus on the IM validation process applied to the reference IM. Additional time was spent between meetings adding definitions to concepts, searching for evidence, and consulting with SMEs, with a rough estimate of 1 h/wk. The herculean effort of keeping 10 organizations engaged over time was facilitated by the requirement to report progress twice a year to the NKBDS Steering Committee and Workgroup leaders. Also the use of Guiding Principles helped the team to stay focused and make progress. These are shown in Table 1.

Table 1.

Information model process guiding principles

Concepts represent core documentation for inpatient or acute care settings. The focus is on staff nurse generalist documentation. Documentation exists in flowsheets and does not include concepts located elsewhere in the EHR (problem lists, care plans, etc.). The overall goal is to reduce documentation by removing repetition or duplication. The focus is on the information (data) and not how the concepts are built in the EHR. Give everyone a chance to voice their opinion, use consensus to move forward, and acknowledge that some information may not be included in the final model. Don’t get stuck reworking previous content thinking everything has to be perfect. Discussions are a balance of clarity and forward movement.

EHR: electronic health record.

The reference GU IM was modified to include the EHR concepts of each organization and current evidence-based guidelines.^27–31 LOINC and Systematized Nomenclature of Medicine-Clinical Terms (SNOMED CT) were engaged for existing concepts and value sets, respectively. As neither terminology includes concept definitions, IM definitions were resourced from the literature and medical dictionaries.^32–34

The team consulted SMEs within member organizations about the content of the evolving model to ensure the GU IM captured the essential GU data for documentation; 4 team members conferred with staff nurses who provide direct, hands-on patient care. Staff were asked to verify coverage of proposed concepts and recommend additions or deletions. The GU team member reported informal results back to the team on subsequent calls. One participant consulted with a GU specialist and researcher who identified some gaps in interventions and devices, with the overall recommendation to engage staff nurses in the review. Three organizations received feedback from staff nurse SMEs that included agreement on the use of GU within defined limits. An organization may not have used some concepts such as UI type, UI pattern, and UI duration; however because these items were used by others organizations, the consensus was to keep them in the IM. Recommendations were made to keep items such as the value set for urine color, and to combine concepts such as female, male, and transgender genitalia into one concept called genitalia characteristics. Finally, value sets were converted from brand names into generic terms (eg, Foley was changed to urinary catheter).

Design

The design of an IM focuses on the organization and structure of identified concepts with definitions, their relationship, and data characteristics. It requires an iterative process of identifying content, and specifying data elements, their relationship, and data characteristics. Guidelines for model design include general rules, best practices, and common agreements¹⁴ based on existing clinical IMs and domain terminologies.^12,17 A consensus process for the inclusion or exclusion of data elements is frequently used without specification of a formal process.^9,14,15,22 There are different levels of specifying data characteristics, ranging from simply naming a concept with definitions¹⁹ and value sets, to inclusion of the data type,^9,12,19 data constraints, and cardinality.^2,9,12,15 The phase of mapping IM content to standardized terminologies can be part of the initial modeling process or conducted later. A few IMs map or plan to map data to SNOMED CT,^16,20,23 LOINC,^5,9 the Unified Medical Language System,²¹ or RxNorm.²¹

Application

The design of the GU IM was depicted in both a Unified Modeling Language (UML) diagram and fully specified in Excel (Microsoft Corporation, Redmond, WA). The design consists of concepts, definitions, value sets, component relationships, and data characteristics. Figure 2 shows the overall GU IM concepts and relationships.

Figure 2.

Genitourinary (GU) system information model concepts and their relationships. AV: Arteriovenous fistula; ML: Milliliter; S&S: Signs and Symptoms; UI: urinary incontinence; WDL: Within Defined Limits.

The team reviewed the reference GU IM which included 8 classes or panels and 36 concepts. The Nursing Process was the framework for the relationship between the concepts: assessments, nursing diagnoses, interventions, and outcomes.³⁵ A draft UML diagram was developed showing the classes, concepts, and relationships between the various IM classes.³⁶ While all participating organizations used nursing diagnoses, the concept was captured in care plan functionality and not in EHR flowsheets. Therefore, nursing diagnoses were depicted in Figure 2 with a connecting line between assessments and interventions. Outcomes were the result of reassessments performed over time, so no unique outcome concepts were identified. Interventions were displayed within GU categories, though implementation may differ in EHRs with a single list, rather than a hierarchical categorization. Synonyms were identified for multiple terms used for the same concept within or across organizations. Finally, data characteristics specified were concepts, definitions, synonyms, data type (eg, nominal, ordinal), and the associated value sets. Multiple iterations of the revised IM between the design step and the validate step occurred.

Validate

Validation of an IM is the process of coming to consensus about the content and coverage of an IM across organizations. A few methods of external validation are conducting Delphi rounds with SMEs,⁹ obtaining face validity by experts;¹² comparing IMs with patient data,¹² testing with use cases,²¹ evaluating logical consistency in Protégé,²¹ and identifying errors, inconsistencies, absences of information, or misleading specifications.¹⁴

Application

After an initial revision of the reference IM, comparing it to current practice, the literature, and data standards, the team mapped their organization’s flowsheet metadata to the model. FloMap,³⁷ which is a secure, Internet-based mapping software developed for the project, was used for the mapping. FloMap uses Boolean logic searching to find local data that matched the GU concepts as shown in Figure 3.

Figure 3.

FloMap flowsheet mapping tool example of Boolean searching.

Each row from the metadata consists of a unique ID; description, name displayed; template name (data entry screen) and data grouping within the screen; the number of observations, encounters, and patients; and first and last concept use dates. The metadata represents the actual documentation by clinicians from all departments at each organization. If available, all organizations provided flowsheet metadata used in the Epic EHR (Epic Systems, Verona, WI) and a SQL script that allowed each organization to extract the metadata in the same manner and upload the metadata to a secure server. Definitions for concepts were critically important for the mapping to ensure that team members had the same understanding when searching and matching their data. FloMap also provided reports comparing mappings across all of the organizations. The reports were used by the team to reach consensus about concept retention or augmentation. Updates to the UML diagram were made throughout this process from multiple iterations until completion of the final diagram. Additional reference IM details can be found in a previous publication.⁵ A report comparing value sets across organizations was subsequently created to identify which value set choices to include for each IM concept. A comparison of GU terms with standardized terminologies was also completed to determine missing concepts and wording. Individuals interested in the use of FloMap who join the work group have access to the software.

Several methods were used to refine the process for validation of the GU IM. In addition to mapping data from multiple organizations and a consensus-based approach, SMEs were included early in the GU IM process and in the validation process. After completing the model, an Internet-based survey was piloted with the team. The team was asked to discuss with staff nurses to determine agreement with the panels, concepts, definitions, and value sets. Participants could recommend modifications, additions, or deletions to the model. Following the pilot, a convenience sample of 19 organizations was recruited through the American Nursing Informatics Association whose predominantly nurse informaticians membership is in practice settings. The participants were asked to consult with staff nurses or nursing GU experts prior to completing the survey and compare the pilot IM with their EHR. There was 85% or higher agreement for all panels and all concepts, with the exception of 77% agreement for UI, requiring concept clarification of UI onset. Consensus was used to reconcile all survey recommendations by a research team subset. The UI pattern concept was divided into UI frequency and UI occurrence. One preexisting insertion date (of a urinary catheter) concept was added and additional values for GU IM concepts included: device type (suprapubic, Coudé, triple lumen, temperature monitoring urinary catheter). Values removed included “insertion difficulty,” considered a complication rather than a reason to continue a urinary catheter. Numerous synonyms were identified for existing values, such as urinary catheter and indwelling catheter. Following reconciliation of survey recommendations, the final GU model was reviewed by the team. The GU IM is composed of 6 panels, 38 concepts with definitions, and 232 values. The final GU IM detail is included in Supplementary Appendix 1.

Disseminate

Disseminate is spreading information about the work and results of validating IMs; it is a continuous process from the beginning through sharing results and future evaluations. Distributing information about and results of validating IMs commonly is done through publications and presentations to facilitate use of the work through informatics organizations such as the American Medical Informatics Association (https://www.amia.org/), Healthcare Information and Management Systems Society (https://www.himss.org/), and Alliance of Nursing Informatics (https://www.allianceni.org/), or inclusion in Health Level Seven (HL7)² or a repository.¹⁴ The practical use of IMs by others also can be done through the development of artifacts such as EHR screenshots or templates, database designs, or examples of reports from use of the data.^2,14,19

Application

The research team is one of the workgroups, accountable biannually for reporting goals and progress to the Steering Committee and other workgroup leaders. Additionally, publications and presentations shared plans and results. Two unique methods of dissemination are in process: dissemination through a group in LinkedIn, Big Data: Empowering Health (https://www.linkedin.com/groups/12096820/), and through the eRepository (in process) focused on dissemination of best practices (http://www.nursingbigdata.org/). The GU IM will be encoded by the NKBDS Encoding and Modeling Workgroup using LOINC and SNOMED CT codes. When necessary, new codes will be requested from LOINC and SNOMED CT consistent with the Office of the National Coordinator’s Interoperability Standards Advisory recommendations.³⁸

DISCUSSION

A refined IM validation framework (Figure 1) was developed and applied to standardize the IM process for the validation of a set of core data elements, definitions, and value sets to document nurse assessments, interventions, and care outcomes using flowsheet data from multiple organizations. In the refined IM, clinical nurse input from SMEs was essential for model usefulness and validation. The SME input was leveraged early in the process to ensure relevance of defined concepts to clinical care. Further, obtaining SME input from multiple organizations using an Internet-based survey proved valuable to increase the generalizability of the revised IM data elements and value sets applied to a GU case.

The standardization of nurse-sensitive data is important for several reasons. Nurse-sensitive data captures the delivery of care with evidence-based guidelines. “As a profession, nursing has a responsibility to measure, evaluate, and improve the quality of nursing practice.”³⁹Nurse-sensitive indicators can be applied to the assessment of care and quality improvement purposes.^28,39–41 The CDMs, such as OMOP and PCORI, include clinical and claims data that are standardized and primarily represents physician and advanced practitioner’s care; the addition of nurse-sensitive data such as catheter-associated urinary tract infection and UI assessments and interventions would enrich these databases to more precisely target care for better outcomes. Current with implementation of IMs, nurse-sensitive data extracted from real-world data in EHRs can extend standardized content for more robust precision health with big data analytic strategies to improve an understanding of complex interactions of physiological, psychosocial, environmental, and lifestyle factors.⁴²

The value of the updated framework for validation of IMs can be extended to additional topics, such the 10 reference models developed by Westra et al.⁷ The NKBDS IM work group previously completed the pain IM and is currently collaborating on the validation of IMs for falls prevention and venous thromboembolism prevention. The next model will be the nursing admission history, which aims to significantly reduce the burden of nursing documentation. Chow et al¹³ previously developed a nursing IM for hospital acquired pressure ulcer prevention. Additional IM topics for development or validation will focus on nurse-sensitive measures as defined by the National Quality Forum (https://www.qualityforum.org/Publications/2004/10/National_Voluntary_Consensus_Standards_for_Nursing-Sensitive_Care__An_Initial_Performance_Measure_Set.aspx) or the Nursing Database of Nursing Quality Indicators (https://www.pressganey.com/resources/program-summary/ndnqi-solution-summary).

There were several limitations identified in revision of the IM validation process. Often, the evidence for commonly used nurse sensitive data is inconsistent because of the disparity in definitions used and methods of capturing the data. The IM specifies key concepts and value sets but does not identify the most efficient and effective way to represent nursing care in a documentation system. Participating organizations experienced some challenges during the data extraction process within their organizations such as a recent change in vendors, policies of organizations about sharing data outside their organizations, and the time of technology staff to pull the data. These limitations also impacted the IM development timeline of 23 months.

FUTURE IMPLICATIONS

Examples and research on the application and implementation of the flowsheet IM is required to quantify nursing care for influence on patient outcomes. Strategies are needed to support vendors and organizations to identify how these standardized clinical concepts can be utilized in their databases. Actual use of semi-structured nurse-sensitive data in IMs is essential to assimilating nurse data in a repository and optimizing interoperability. To date, the IM is not implemented yet until it is disseminated publicly. Each organization has components of the model in their existing system, but work is needed yet to gain consensus within organizations to implement the reviewed GU model. Another future activity is finalizing a process for updating the IMs; we have discussed keeping a list of requests and updating the IMs every 3-5 years.

CONCLUSION

We developed a process framework for validating IMs to increase interoperability of nurse sensitive data across organizations including application to clinical decision support, care coordination, quality improvement, and research. The IM process refinements, adding evidence and SMEs, produced a clinically relevant and evidence-based model of GU care using EHR flowsheet data. Future IM validation work will continue process refinement to increase efficiency and generalizability.

AUTHOR CONTRIBUTIONS

All authors substantially contributed to the design; acquisition, analysis, or interpretation of data for the work; and drafting or critical revision the manuscript and final approval of the version, and agreed to be accountable for all aspects of the work.

SUPPLEMENTARY MATERIAL

Supplementary material is available at Journal of the American Medical Informatics Association online.