Automatic population of eMeasurements from EHR systems for inpatient falls

Abstract

Objective

Representing nursing data sets in a standard way will help to facilitate sharing relevant information across settings. We aimed to populate nursing process and outcome metrics with electronic health record (EHR) data and then compare the results with event reporting systems.

Methods

We used the “eMeasure” development process of the National Quality Forum adopted by the American Nurses Association. We used operational definitions of quality measures from the American Nurses Association and the US Institute for Healthcare Improvement and employed concept mapping of local data elements to 2 controlled vocabularies to define a standard data dictionary: (1) Logical Observation Identifiers Names and Codes and (2) International Classification for Nursing Practice. We assessed feasibility using the nursing data set of 7829 and 8199 patients from 2 general hospitals with different EHR systems. Using inpatient falls as a use case, we compared the populated measures with results from the event reporting systems.

Results

We identified 17 care components and 118 unique concepts and matched them with data elements in the EHRs. Including suboptimal mapping, 98% of the assessment concepts mapped to Logical Observation Identifiers Names and Codes and 52.9% of intervention concepts mapped to International Classification for Nursing Practice. While not all process indicators were available from event reporting systems, we successfully populated 9 fall prevention process indicators and the fall rate outcome indicator from the 2 EHRs. We were unable to populate the falls with an injury rate indicator.

Conclusions

EHR data can populate fall prevention process measure metrics and at least one inpatient fall prevention outcome metric.

Introduction

The rapid implementation of electronic health records (EHRs) and the integration of EHR data into clinical data repositories (CDRs) means that large quantities of clinical data are now available. Combining these data with computational technologies shows promise in improving nursing knowledge representation and patient outcomes.¹ In an effort to align quality indicators with codified clinical and billing data, the US Centers for Medicare and Medicaid Services converted 113 National Quality Forum (NQF)-endorsed Clinical Quality Measures from a paper-based format relying on chart abstraction into an electronic “eMeasure” format.² For nursing-sensitive quality indicators, little is known about the feasibility of using nursing data in EHRs to populate the eMeasure format as well as for benchmarking across disparate EHRs.

Nonstandardized data are used in most EHR systems due to the lack of knowledge regarding standardized processes and data among the nursing executive and clinician workforce.¹ However, multiple years of experience with EHR systems creates an opportunity to explore ways to standardize locally developed nursing vocabularies. Our previous nationwide survey of EHR systems conducted in 2012 revealed that more than half of general hospitals in Korea implemented a local nursing vocabulary.³ We hypothesized that if nursing data in these local systems were represented in a standard way, they could be shared and compared on regional and national levels and would facilitate the identification of trends and outcomes. Here we describe the process of transforming local nursing data into a standardized format to populate fall prevention quality measures, and demonstrate the feasibility of this method within 2 disparate EHR environments. We also compared this approach with the existing manual approach using standard event reporting systems.

Methods

Site and Settings

Two hospitals located in metropolitan areas of Korea with different locally developed EHR systems participated in this study. One was a tertiary academic hospital with 900 beds and the other was a secondary hospital with 800 beds. Both hospitals used EHR systems for >8 years that included nursing vocabularies with both local terms and nursing statements that were based on the International Classification for Nursing Practice (ICNP) and North American Nursing Diagnosis Association, Nursing Intervention Classification, and Nursing Outcomes Classification (3N) controlled vocabularies.

Identification of Outcome Metrics for Inpatient Falls

The eMeasure development process of the NQF adopted by the American Nurses Association (ANA) and the following National Database of Nursing Quality Indicator outcome metrics for patient falls were used to aggregate local data and to populate quality measures: patient fall rate and patient falls with injury rate.⁴ For process metrics, we used the Institute for Healthcare Improvement definitions.⁵ Process indicators measure compliance with key care components of evidence-based nursing. Improvements in the scores for individual process measures indicate that the processes underlying these care elements have improved, and improvements in patient outcomes should follow. Mant⁶ advised using process metrics to measure the quality of health care if linkages exist between given processes of care and patient outcomes. Where such measures are available, relevant, and practical, they are used in conjunction with outcome metrics since they are sensitive to differences in the quality of care in the short term; typically, before outcome measures are available.^6,7

Identification of Standard Content of Fall Related Care

We formed a project team consisting of experts in the following relevant domains: nursing informatics, terminology, quality management, and patient safety. Two College of Nursing professors, 4 graduate students with 5–10 years of practical experience at university hospitals, nursing staff, and managers of the sites were involved. We used the International Classification for Patient Safety’s (ICPS) conceptual framework to identify patient safety concepts for falls.⁸ The process of how standard content was identified from a literature review, and arranged into key care components under the ICPS framework, is described in detail elsewhere.^9,10 The aim of the literature review was to identify evidence-based nursing content related to falls that could be mapped to nursing documentation in the EHR systems through standard terminologies. We identified 17 key care components corresponding to fall prevention and post-fall nursing care. We also searched national and regional nurse-sensitive quality databases with the aim of supporting quality assessment and demonstrating the nursing contribution to patient care using measures similar to those proposed in the ANA initiative in 1995¹¹: National Database of Nursing Quality Indicator,¹² California Nursing Outcomes Coalition,¹³ Military Nursing Outcomes Database,¹⁴ and VA Administration Nursing Outcomes Database.¹⁵

Data Collection

We performed the following 4 data collection steps: (1) review of practice guidelines and literature to identify fall prevention key care components, process, and outcome measures; (2) data mapping from local nursing vocabularies to standardized terminologies; (3) populating eMeasures with standardized data; and (4) comparing an eMeasurement system with an event reporting system in terms of its ability to generate process and outcome metrics. Figure 1 shows the relationships between these steps.

Figure 1.

The 4 steps of this study procedure. LOINC: Logical Observation Identifiers Names and Codes; ICNP: International Classification for Nursing Practice.

Identifying fall prevention key care components and metrics

We selected the 7 guidelines or tools recommended by The Joint Commission in 2015 and the Korean Hospital Nurses Association Nursing Practice Guidelines to identify key care components and fall prevention measures.^16,22Figure 2 displays the list of resources.

Figure 2.

Inpatient fall prevention practice guidelines and tools reviewed.

• Agency for Healthcare Research and Quality Fall Prevention Toolkit¹⁷
• ECRI Institute Falls Prevention Guidance¹⁸
• Institute for Clinical Systems Improvement Prevention of Falls (Acute Care) Protocol¹⁹
• Institute for Healthcare Improvement's (IHI) Transforming Care At The Bedside How-to Guide⁵
• The Joint Commission Center for Transforming Healthcare's Targeted Solutions Tool (TST^®) for Preventing Falls²⁰
• Veterans Affairs. VA National Center for Patient Safety: Falls Toolkit²¹
• Veterans Affair (VA) National Center for Patient Safety Implementation Guide For Fall Injury Reduction²²
• Korean Hospital Nurses Association Nursing Practice Guidelines of Prevention of Inpatient Falls²³

Outcome and process metrics and the corresponding care components to be extracted from each EHR system were vetted with inpatient nursing teams. The calculation of outcome indicator logic followed the ANA/National Quality Forum endorsed guidelines for data collection.²⁴

Data mapping to standardized nursing terminologies

The standard fall prevention content consisted of nursing assessment, diagnosis, and intervention activities. We refer to nursing data as the data set in EHR systems including 3 broad categories of elements: (1) nursing care; (2) patient or client demographics; and (3) service elements. Data elements are the atomic units of nursing data. For example, when evaluating cognitive function, specifically delirium, several descriptive terms exist in standard content or EHR systems including altered mental status, confusional state, and delirium. We designate delirium as one data element. We mapped data elements to standard terminology concepts from Logical Observation Identifiers Names and Codes (LOINC) for administrative, assessments, and diagnoses, and ICNP for interventions. The Health and Welfare Ministry of Korea endorses LOINC and ICNP as national healthcare standards. We used Regenstrief LOINC Mapping Assistant (RELMA, LOINC browser) and the ICNP Browser at the International Council of Nursing. Two graduate students with 5 and 10 years of clinical experience conducted the initial mapping, then the authors (IC and EHB) reviewed and discussed further to classify the mapping results into 3 categories of optimal (exact match), suboptimal (partial match), and not mapped (no match). We used the concept ontology of the Systematized Nomenclature of Medicine-Clinical Terminology (SNOMED-CT) as a sematic map indicating a type of graphic organizer that visually represented relationships among categories of concepts. We also used the 8 guidelines of preventing falls for assuring the relationships among concepts in the context of fall prevention. We classified the cases into the optimal mapping, in which a data element completely corresponding to a concept semantically or in relation to an “is a” (parent or child relation) with a concept. We classified discrepancies or disagreements on semantics between reviewers as suboptimal.^25,26

The data dictionary for each hospital guided the collection and normalization of data from multiple sources: the EHR systems, reporting systems, and administrative databases. Table 1 shows the scheme of the data dictionary and data examples. The EHR data included both structured and semi-structured formats. We coded nursing statements in nursing notes in a semi-structured format.

Table 1.

Scheme of Inpatient Fall Data Dictionary and Data Examples

Field	Example of assessment	Example of intervention
Care component	Cognitive factors	Impaired mobility
Care type	Assessment	Intervention
Care activity	Observe (assess or evaluate) delirium/confusional state/altered mental status/ reversible dementia/organic brain syndrome	Regularly scheduled assistance with toileting
Data element	Delirium	Aid of toileting
Definition	Confusional state	Regularly scheduled assistance with toileting for patients with impaired mobility
Terminology	LOINC	ICNP Intervention
Concept(s)	Delirium [MDSv3]	Assisting with toileting
Terminology code	54626-7	10023531
Local facility ID	Hospital A	Hospital B
Local EHR ID.version	eChart.2.0	SmartCare.3.0
Local screen-item name	Nursing notes	Nursing notes
Local expression	Observed delirium symptom(degreea: disorganized)	Provide regularly assistance with toileting
Local data type	Assessment statement	Intervention statement
Local code	S03745	A05595

^aAttribute and value of the nursing statement.

Populating falls indicators from EHR data

Based on the data dictionary, we retrieved and collected data from 4 medical-surgical nursing units of each hospital over one calendar year (2014). We used the operational patient selection criteria of NQF measures #0141 (patient fall rate) and #0202 (patient falls with injury) for the targeted medical-surgical units. We included: (1) adult inpatients aged 18 years and older and (2) patients who had been admitted for at least 24 h. We excluded patients with a primary pediatric, psychiatric, or obstetrics diagnosis, or who received resuscitation treatment, or died. We included only patient stays on targeted units during the study period.

Comparing the eMeasurements with event reporting systems

The 2 hospitals independently submitted data from their locally developed event reporting systems. Data consisted of patient demographics, admission information, situational descriptions about falls, post-fall measures, and management information used for root-cause analysis. We compared the completeness of indicators populated with manual event reporting data with indicators populated directly from the EHR systems. We also used these data to cross-check cases recorded in the EHR system and the event reporting system. Three authors (IC, EHB, and SYL) conducted chart reviews to identify fall events hospital managers confirmed new cases.

Data Analysis

The Institutional Review Board at each hospital approved this research project. Each hospital provided de-identified data from the clinical data repository (CDR). To identify the target patients, we applied operational criteria that were modified from 2 ANA endorsed nursing-sensitive outcomes; total falls per 1000 patient days for fall rate, and injury falls per 1000 patient days for injury fall rate.^24,27 We expressed process indicators as proportions or average frequencies, such as the number of nursing activities conducted to assess a patient’s disease-related factors, cognitive factors, behavior factors, or therapeutics during the days when the patient was at risk of falling.

Results

Identifying Fall Prevention Key Care Components and Measures

Figure 3, the Fall Prevention Indicator Map, includes the key fall prevention care components and measures identified through literature and guideline review. The indicator map represents the structural relationships between fall prevention key care components and indicators. The left side of the figure displays the parts of the care components used to calculate the indicators that appear on the right side of the figure. The figure includes 2 outcome indicators and 9 process indicators.

Figure 3.

Indicator map of inpatient falls (^aoutcome indicators, ^bprocess indicators).

Data Mapping to Standardized Nursing Terminologies

The concepts of nursing assessments and fall incident categories were all successfully mapped to LOINC except the concept of injury level in incident characteristics. According to its definition, fall injury level has 4 categories in its value list: none, minor, moderate, and major. However, we mapped this concept to the LOINC concept event outcome of the Medical Event Reporting System—Total HealthSystem taxonomy, for which the value list was 4 categories: no recovery with harm, no recovery with no harm, near miss with unplanned recovery, and near miss with planned recovery. The post-fall assessment concepts mapped to LOINC except the following 2 concepts: absolute bed rest state and close observation state.

A total of 52.9% of nursing intervention concepts mapped to ICNP (including suboptimal mappings). For example, we successfully mapped the concepts observation and surveillance, provision of protocol, and post-fall care provided to 18 intervention statements in ICNP. However, we could not find ICNP interventions on the nursing activities, communication and education such as sharing high risk patients and use of visual fall risk indicator. The closest interventions were the collaborating with interprofessional team and fall prevention in ICNP. We also found almost all of the universal precaution nursing activities and environmental interventions were related to the ICNP interventions, fall prevention and environmental safety management. However, the ICNP interventions were too implicit and abstract. We classified those into the not mapped. Tables 2 and 3, respectively, present examples of the mapping results for LOINC and ICNP intervention statements. Each data element is used to apply the inclusion or exclusion criteria, compute denominators, or aggregate by the care component. The full version of mapping list is online in Supplementary Material S1.

Table 2.

Example Nursing Assessment Data Elements Whose Concepts Mapped onto LOINC

Care component	Data element	LOINC concepts (ID)	Property	Time	System	Scale	Method	Relation with populating indicator
Reason for encounter	Primary diagnosis	Primary diagnosis (18630-4)	Imp	Pt	Patient	Nom	—	Inclusion criteria
Administrative	Length of stay	Hospital stay duration (78033-8)	Time	Pt	Patient	Qn	—	Patient days
Pathophysiologic disease-related factors	Use of anticoagulants/coagulopathy diagnosis/the first day of surgical procedure	Bleeding risk [CCC] (81225-5)	Find	Pt	Cardiovascular system	Ord	Observed.CCC	Frequency of nursing assessment of risk factors
	Nursing diagnosis: visual impairment	Visual alteration [CCC] (28242-6)	Find	Pt	Eye	Ord	Observed.CCC
Cognitive factors	Delirium	Delirium [MDSv3] (54626-7)	–	Pt	Patient	–	MDSv3
	Consciousness	Level of consciousness (80288-4)	Find	Pt	Patient	Ord	–
Behavior factors	Fall history	Fall history on admission [MDSv3] (54849-5)	–	Pt	Patient	–	MDSv3
Therapeutic factors	IV line	Intravenous apparatus (59457-2)	Find	Pt	Patient	Ord	Morse Fall Scale
	Foley catheter	Indwelling catheter [MDS] (45628-5)	Find	Pt	Patient	Ord	MDS
Incident characteristic	Injury level	Event outcome [MERSTH] (42545-4)	Find	Pt	Patient	Ord	Observed. MERSTH	Frequency of injurious falls

CCC, clinical care classification; MDS, Minimum Data Set; MERSTH, Medical Event Reporting System Total Healthsystem.

Table 3.

Example Nursing Intervention Data Elements Whose Concepts Mapped onto ICNP

Care component	Data element	ICNP nursing intervention statement	Code	Relation with populating indicator
Universal precaution	Keep floor surfaces clean and dry	Fall preventiona	10040211	Frequency of provision of universal precaution
	Make sure patient’s essential needs are within easy reach	Fall preventiona	10040211
Communication	Use visual indicator	Fall preventiona	10040211	Frequency of provision of communication and education interventions
	Notify or take over risk information between clinicians	Collaborating with interprofessional team	10039416
Observation and surveillance	Use bed or chair alarm	Maintaining fall safety alarm	10041525	Frequency of provision of observation and surveillance interventions
	Assess fall risks (using a tool)	Assessing risk of falls	10023520	Counting use of fall risk tool
Provision of protocol	Provide safety aids	Providing safety device	10024527	Frequency of provision of protocol interventions
	Provide scheduled assistance with toileting	Assisting with toileting	10023531
	Offer help using the toilet
Provision of education	Educate high-risk patient/family	Teaching family about fall prevention	10040269	Frequency of provision of communication and education interventions
Environmental intervention	Rearrange room to clear paths	Environment safety managementa	10042507	Frequency of provision of environmental interventions
	Adjust bed into locked position
	Secure raised toilet seat to commode
Post-fall care provided	Apply ice pack	Applying cold pack	10036468	Frequency of nursing assessment of risk factors
	Perform neurologic assessment periodically	Monitoring neurologic status	10035326
	Notify physician and other nurses about a fall	Collaborating with interprofessional teama	10039416

^aMeans the not mapped category.

The data model represented in Tables 2 and 3 includes 84 LOINC nursing assessment concepts and 22 ICNP intervention concepts. We mapped 43 (51.2%) and 15 (68.2%) of those to local data in 1 EHR, and 37 (44.0%) and 15 (68.2%) of those for the other EHR. We classified 5 of the 15 intervention mappings as suboptimal. Some data elements comprised several data item-value sets, such as fall risk assessment tools and regular rounding. An array of data items comprised the fall risk assessment tool: total score, sub-items and scores, and risk status. The regular rounding as a fall prevention intervention included several universal precaution activities, such as keeping hospital bed brakes locked, educating on use of call light, keeping the patient’s personal possessions within reach, and assessing the need for 1:1 monitoring. The fall prevention education provided on admission was also a composite intervention. Patients’ personal risk factors determined the other interventions provided.

Populating Falls Indicators from EHR Data

Data from 7829 (73 525 patient days) and 8199 (58 530 patient days) patients admitted during the research period from 2 hospitals met the inclusion criteria. Using the 2 data sets from EHR systems, 10 of the 11 indicators were successfully populated; the exception was the rate of falls with injury (see Table 4). The fall rates were computed 0.71 and 0.41 per 1000 patient days, respectively (see Table 2 for data elements used to populate the outcome indicator). The falls with injury rate was calculated at 0.27/1000 patient days at only one hospital. It was calculated from 2 data elements of the length of stay and the injury level. Coded nursing assessments, diagnoses and interventions recorded in each EHR system populated the 9 process indicators (see examples in Tables 2 and 3).

Table 4.

Comparison of Outcome and Process Indicators in EHR systems, Event Reporting Systems, and the Combination of These 2 Systems

Outcome and process indicators on fall prevention	Hospital A			Hospital B
	EHRs	Event report	EHRs + event report	EHRs	Event report	EHRs + event report
Fall ratea (falls per 1000 patient days)	0.71	0.79	1.32	0.41	0.02	0.43
Injury fall ratea (falls with injury per 1000 patient days)	NA	0.27	NA	NA	NA	NA
Patients assessed using a fall risk tool within 24 h of hospital admissionb (%)	88.29	NA	88.29	96.98	NA	96.98
Average frequency of using a fall risk assessment toolb	0.44	NA	0.44	0.55	NA	0.55
Patients with a nursing assessment of risk factors within 24 hb (%)	94.44	NA	94.44	89.9	NA	89.9
Patients with a daily nursing assessment of risk factorsb (%)	1.29	NA	1.29	1.0	NA	1.0
Patient days when there was a risk of falling (%)	16.97	NA	16.97	21.0	NA	21.0
At-risk patients to which universal precautions were applied within 24 h of risk identificationb (%)	99.78	NA	99.78	0.87	NA	0.87
At-risk patients to which communication and education interventions were applied within 24 h of risk identificationb (%)	99.22	NA	99.22	10.9	NA	10.9
At-risk patients to which observation and surveillance interventions were applied within 24 h of risk identificationb (%)	78.81	NA	78.81	0	NA	0
At-risk patients to which provision of protocol interventions were applied within 24 h of risk identificationb (%)	13.42	NA	13.42	6.0	NA	6.0
At-risk patients to which environmental interventions were applied within 24 h of risk identificationb (%)	92.75	NA	92.75	89.1	NA	89.1

^aOutcome indicators and ^bItalics process indicators. Abbreviation: NA, not available.

Comparing the eMeasurements with Event Reporting Systems

None of the process indicators and only the fall incident outcome indicator was available from the reporting data (see Table 4). The falls with injury rate were not always specified due to incomplete data reported in incident reports. The variables of Table 4 are outcome and process indicators, and a denominator for process indicators to be applied within 24 h of risk identification.

We found patient falls recorded in the EHR systems, but not reported in the incident reporting systems. These cases corresponded to 39 (40.2%) out of 97 incidents for one hospital and 24 (96%) out of 25 incidents for the other hospital. Through the chart review, we found 45 (46.4%) and 1 (4%) cases recorded in the incident reporting system, but not recorded in the EHR systems. Figure 4 illustrates the summary of these results using Venn diagrams.

Figure 4.

Comparison of falls detection results in EHRs and event reporting systems.

When we combined the fall incidents from EHRs and event reporting systems, the fall rates were 1.32 and 0.43, respectively.

Discussion

In the present study, we successfully populated nursing quality indicators from EHR systems in a format comparable with those used at other facilities. This was possible due to the availability of standard fall prevention content, standard terminologies, and a mapping process completed by experienced informaticians and nursing experts. Despite the challenges in mapping some nursing interventions to ICNP, we successfully autopopulated all the process indicators. In addition, using EHR data we detected fall events that were not available in the event reporting systems.

LOINC had excellent content coverage for the 10 nursing assessment care components needed to populate the fall prevention indicators. The care components contain data elements that were measurements or observations captured in nursing assessments. This result demonstrates higher concept coverage than found in an earlier study²⁸ which mapped 294 concepts extracted from nursing flowsheet documents of 6 organizations onto SNOMED CT and LOINC. Here, they reported mapping rates from 70% to 82%. Some of the data in our study comprised nursing judgments in diagnoses such as visual alteration, auditory alteration, and physical mobility impairment. These concepts mapped to precoordinated concepts from nursing terminology or assessment tools such as the Clinical Care Classification²⁹ or the Morse Fall Scale³⁰; both of which are incorporated in LOINC. Our findings imply that LOINC includes acceptable content coverage of nursing judgments (e.g., findings) related to falls prevention. LOINC is a catalog of measurements including laboratory tests, clinical measures such as vital signs, anthropomorphic measures, and standardized survey instruments. This finding suggests that LOINC can also be used as a standard terminology to represent fall prevention nursing assessment and observation concepts.³¹

As for intervention mapping, a relatively small number of ICNP intervention statements matched the nursing intervention data elements. Most universal precaution and environmental intervention care components were not mapped to ICNP interventions. The root cause is the low levels of granularity at which the precoordinated intervention statements in ICNP are aggregated. For example, the intervention statements of sharing/communication high risk patients between staff, use of visual indicator, keep family sitter, and communicate patient’s history, risk factor, treatment, and plan all corresponded to the single concept, collaborating with interprofessional team. Other statements imply various sets of activities that a nurse could provide to patients, such as assessing risk for falls on admission, fall prevention, and post-fall evaluation. The expressions do not provide detailed information about the kinds of activities planned and provided in practice; but rather as an abstract representation of relevant activities. The ICNP’s level of abstraction restricts the accuracy of concept mapping and secondary use to represent practice, quality reporting and research. However, ICNP has current mappings to SNOMED CT and this hierarchy could be leveraged to provide more detailed child concepts.³² Since the ICNP version 2 release, the ICNP terminology team has developed terminology subsets that are both clinically relevant and user-friendly. They encourage the development and maintenance of ICNP catalogs or subsets as part of the ICNP terminology life cycle, if such catalogs will enhance the usability of the terminology in practice. The ICN has reported that a pressure ulcer catalog is under development for the patient safety domain. However, there is no explicit fall prevention catalog to represent activities to prevent patient falls, even though inpatient falls are widespread, a common nursing-sensitive outcome, and a serious threat to patient safety. Our research team previously notified the ICNP terminology team of the importance of an inpatient fall prevention catalog and commenced its development using the conceptual ICPS framework.¹⁰

Unlike the event reporting systems, our EHR-based approach allowed us to populate process indicators as well as outcome indicators across multiple time periods. Using the existing event reporting system, we could extract outcome indicators over a relatively limited period. The incident reports did not have complete nursing process data. Considering the utility of process indicators in the short term for improving the quality of care⁶ and the fact that inpatient falls are relatively rare event (e.g., occurring to 0.05%–0.3% patients^33,34), it is a great advantage that EHRs are able to compute process indicators. Process indicators have more power to detect real differences in the quality of nursing care provided each day. The ability to compute process indicators from EHR data supports the feasibility of using eMeasures for internal quality improvement activities, as well as for national reporting.² The eMeasure approach has the potential to reduce the nursing management burden associated with manual reporting and increase efficiency in monitoring.

One unexpected finding of the present study was the detection of new fall incidents from EHR data that had not been reported via the event reporting systems. The validation process detected several false-positive cases, but true-positive cases increased the percent of falls identified at the 2 hospitals by 40% and 96%, respectively. This finding confirmed empirically the under-reporting of safety incidents in the event reporting systems.^35,36 This discovery implies that the use of EHRs in outcome measurement offers a new opportunity to address under-reporting. To achieve more accurate patient falls outcome reporting, we may need to redesign the reporting process. Use of EHR data to supplement event reporting system data is one option. However, omissions of fall event recording in EHRs is not straightforward. Some nurses may forget to document a fall. Another possibility is that nurses avoid entering the same content into 2 different systems (e.g., double documenting). A study^37,38 investigating the accuracy of nursing notes in 6 EHR systems using a time-motion method, reported 85%–92% matching rates between the activities observed and recorded. This report supports the possibility of the later. Redesigning the incident reporting process to include reporting incidents solely in the EHR system, could lead to more complete data capture.

One of the strengths of this study is that the participating hospitals independently implemented EHR systems that cover all inpatient nursing documentation. These systems have been in place for >8 years. Our analysis relied on the hospital CDRs, not a data mart. This made it possible to access any form of data (e.g., structured, semi-structured, and free text) without missing any data included in the live EHR systems. Another strength is the fact that we populated eMeasures using semi-structured data representing nursing process information. To our knowledge, this was not done previously. One study³⁹ analyzed flowsheet data documented in multiple EHR systems and created 14 nursing information models based on clinical topics. It defined a flowsheet as a format for facilitating structured data capture and as a template that can be specific to a unit or discipline. They found that flowsheets contain groups of related assessments and interventions, and individual items for actual documentation. Another study⁴⁰ analyzed flowsheet data of 2 million patients and developed an ontology to support secondary data use. However, the study is based on data documented at a single site in an Epic Systems Corporation EHR (single vendor).

One limitation of this study is the generalizability of our findings to other measures and hospitals. However, considering the characteristics of inpatient falls, we believe that the data elements and processing methods will apply to other preventable adverse event quality measures such as pressure ulcers, pain management, and catheter-associated urinary tract infections. Based on ANA’s criteria for selecting quality indicators for conversion to eMeasures, inpatient falls is a strong candidate. The generalizability of our results to other care delivery systems and EHR systems is unknown. However, the approach to extract process and outcome indicators in a standardized and comparable format can be leveraged with any EHR system. Given that most EHR systems are not based on standards that enable semantic interoperability, the approach described could contribute to the benchmarking of nursing performance across sites and settings.

Conclusions

This study demonstrated the data from EHR environments can generate fall prevention process measures and at least one outcome metric. This supports sharing and comparing nursing-sensitive process and outcome metrics for multiple purposes, including the secondary use of a clinical nursing data set. Although the content and use of standards in EHR systems are incomplete and based on highly variable user preferences, standard nursing terminologies and content can be used to address existing gaps and EHR data can contribute to nursing quality improvement.

Competing interests

None.

Funding

This study was supported by a grant of the Korea Healthcare Technology R&D Project, Ministry for Health, Welfare & Family Affairs, Republic of Korea (No. HI15C1089). IC was also supported by a grant from Inha University (No. 50463-01) and the Hospital Nurses Association of Korean Nurses Association in 2015.

Contributors

IC conceived and designed the study, supervised, and contributed to the data analysis, interpreted results, and drafted and revised the paper. EHB and SYL contributed to study design, data acquisition, and data analysis. PCD provided substantial contribution to the interpretation of data analysis and revised the work critically for important intellectual content.

Supplementary Material

Supplementary material is available at Journal of the American Medical Informatics Association online.