Reliability and Validity of the Checklist for Early Recognition and Treatment of Acute Illness and Injury as a Charting Tool in the Medical Intensive Care Unit

Abstract

Background:

Resuscitation of critically ill patients is complex and potentially prone to diagnostic errors and therapeutic harm. The Checklist for early recognition and treatment of acute illness and injury (CERTAIN) is an electronic tool that aims to provide decision-support, charting, and prompting for standardization. This study sought to evaluate the validity and reliability of CERTAIN in a real-time Intensive Care Unit (ICU).

Materials and Methods:

This was a prospective pilot study in the medical ICU of a tertiary care medical center. A total of thirty patient encounters over 2 months period were charted independently by two CERTAIN investigators. The inter-observer recordings and comparison to the electronic medical records (EMR) were used to evaluate reliability and validity, respectively. The primary outcome was reliability and validity measured using Cohen's Kappa statistic. Secondary outcomes included time to completion, user satisfaction, and learning curve.

Results:

A total of 30 patients with a median age of 59 (42–78) years and median acute physiology and chronic health evaluation III score of 38 (23–50) were included in this study. Inter-observer agreement was very good (κ = 0.79) in this study and agreement between CERTAIN and the EMR was good (κ = 0.5). CERTAIN charting was completed in real-time that was 121 (92–150) min before completion of EMR charting. The subjective learning curve was 3.5 patients without differences in providers with different levels of training.

Conclusions:

CERTAIN provides a reliable and valid method to evaluate resuscitation events in real time. CERTAIN provided the ability to complete data in real-time.

INTRODUCTION

Critical illness continues to be the final common pathway to a multitude of disease processes and is associated with high morbidity, mortality, and resource-utilization.[1] Mortality in the Intensive Care Unit (ICU) continues to be about 20%–30% globally with ICU organization structures being a vital component in mortality prevention.[2] With the increasing recognition of critical illness as a preventable disease state, there is increased thrust on prehospital and early ICU care to influence patient trajectories.[3,4,5] Lessons from emergency and acute care often highlight the vitality of the “golden hour” of resuscitation that gains further prominence in common medical emergencies such as stroke and sepsis.[6] The complexity of critically ill patients makes initial care at point of contact challenging due to the involvement of multiple organ systems and often, the lack of a good history. Critical care physicians have an influx of information from several failing organ systems, that is often processed while making therapeutic decisions and performing life-saving interventions and procedures. This can often result in known or unknown diagnostic errors and delays in health-care delivery, often resulting in increasing financial burden, mortality, and morbidity.[7,8] This is further exacerbated by the intensivist shortage and high nursing and physician to patient ratios that result in multi-tasking and potential for greater diagnostic and therapeutic errors.[9]

The use of standardized checklists has reshaped the approach to situations with a high cognitive burden. The use of checklists has consistently shown increased performance in emergency medicine, trauma, cardiac arrest, and surgeries.[10,11,12] Standardized resuscitation protocols such as advanced trauma life support, basic and advanced cardiac life support, and pediatric advanced life support have contributed to rapidly improving outcomes both globally and nationally. However, the use of checklists in critical illness remains relatively low and unstandardized. Prior work at our center combining the expertise in clinical informatics, critical care, and health-care delivery has resulted in the development of checklists and use of user-friendly interfaces that decrease cognitive burden.[13,14,15] These interventions are intended to decrease diagnostic errors and therapeutic harm while increasing provider satisfaction. The checklist for early recognition and treatment of acute illness and injury (CERTAIN) was developed by investigators at our center as a point-of-care decision-support system.[16] This checklist was developed by an international collaboration using best-practice principles that could be invaluable in low-resource settings with a shortage of trained critical care providers.[3] The core of the CERTAIN tool remains the electronic interface to record and present data to clinicians in real time.

In this study, we sought to evaluate the reliability and validity of CERTAIN charting in real-time at the patient's bedside. Secondary outcomes included charting time, user satisfaction, and usability of the CERTAIN tool.

MATERIALS AND METHODS

Study design

This was a prospective observational study conducted in the 24-bed medical ICU at a tertiary care medical center from July 1, 2015, to August 31, 2015. All adult patients were potentially eligible for inclusion in the study if they were admitted to medical ICU during the hours that at least two of the three physician investigators were present. We excluded patients if they were admitted for monitoring of comfort measures only. Two of the three physician investigators (two critical care fellows [AK, SV] and one internal medicine resident [KMP]) evaluated newly admitted patients that were transferred to the ICU and tracked the initial resuscitation using the CERTAIN tool. The study was approved by Institutional Review Board under waiver of consent due to the observational nature of the intervention.

Checklist for early recognition and treatment of acute illness and injury tool

The CERTAIN tool was designed by consensus opinion and best-practice evidence-based guidelines that are serially updated. CERTAIN provides evidence-based diagnostic checklists, clinical decision support, educational modules on performing critical procedures, and has the ability to time and document real-time interventions (detailed website– www.icertain.org). CERTAIN maybe used as a real-time interface in the care of critically ill patients. Electronic charting function helps to document patient-care related decision-making, diagnostic strategies, and therapeutic interventions in a decision-support format using iterative flowcharts. Two modules, CERTAIN Elite (evaluation of life-threatening emergencies) and CERTAIN Rounds, embedded in the tool would help the clinicians to track and evaluate the life-threatening emergencies of critically ill along with the routine recommended care processes, which needs to be assessed daily for every patient in the ICU. CERTAIN is task-specific, allows for knowledge translation, collaborative workspace, and user-interface style of charting.

The charting component of CERTAIN is an electronic checklist with a user-friendly and intuitive interface [Supplemental Figure 1 ^{(390.8KB, tif)} ].[3,16] The main screen contains basic patient information, such as vital signs, common laboratory parameters, pertinent imaging studies, allergies, medications, problem list (in order of priority), performed interventions, and a timer. The timer evaluates time to intervention (administration of medications, vascular access, start and stop of cardiopulmonary resuscitation, etc.) Clinical decision support and care algorithms are easily accessed from the problem list, procedures, and medication areas. Data are entered either by simple clicks (e.g., value is elevated vs. decreased) or directly entered as numeric and textual values. All data are recorded, and CERTAIN generates a resuscitation record from this information.

Supplementary Figure 1

Checklist for early recognition and treatment of acute illness and injury interface screenshot. The picture shows decision support tool with a card for cardiac arrest. Vital signs and ABCDEs of critical illness are documented in the top, with related information on the sides and cardiac support algorithm in the center. This interface enables quick integration of information on a single page

Outcomes

The primary outcome was reliability and validity of CERTAIN observations. Reliability was defined as an agreement between charting of two CERTAIN observers (inter-rater agreement). Validity was defined as concurrent validity, or agreement, between CERTAIN charting and the current gold standard of electronic medical record (EMR) charting. Reliability was calculated for sub-categories (vital signs, laboratory, and physical examination findings) in addition to overall reliability. The secondary outcomes included feasibility expressed as the subjective ability to perform CERTAIN observations (completed vs. not completed), time to complete CERTAIN charting as compared to the time to complete EMR charting (in minutes), CERTAIN usability (five-points Likert-like scale from 1 for “extremely easy” to 5 for “extremely difficult”) and the learning curve expressed as number of patients to observe to subjectively feel comfortable using CERTAIN interface.

Statistical analysis

Based off a prior pilot study from our center,[17] sample size was calculated using a power of 0.8 assuming previous kappa coefficient of 0.75. The primary outcome was reported as Cohen's kappa coefficients (κ) with 95% confidence intervals. We calculated κ values for inter-observer agreement using CERTAIN, as well as CERTAIN vs. EMR charting. A priori, good agreement was adjudicated as κ ≥ 0.5. Paired and unpaired data computation methods were used to compare two investigators and investigators with EMR, respectively. Continuous data were analyzed using Wilcoxon tests and binomial data using McNemar's test. Statistical significance was set as a P ≤ 0.05. Study data were collected and managed using the REDCap software, and JMP Version 10 (SAS Institute, Cary NC, USA) was used for statistical analyses.

RESULTS

During the 2 months study period, a total of 30 patients were enrolled into this study. Baseline clinical and demographic characteristics are shown in Table 1. There was very good aggregate inter-observer agreement with κ = 0.73. As detailed in Table 2, agreement regarding individual elements was good for vital signs, laboratory tests, and physical examination findings (κ = 0.4–1.0). Agreement on diagnosis was excellent (κ = 0.79). The agreement between CERTAIN users and the EMR chart was good with κ = 0.5, supporting concurrent validity of CERTAIN [Table 3]. The best agreement was found for wheezing (κ = 1.0) and the worst was for temperature and respiratory rate (κ = 0.35 and 0.36, respectively).

Table 1.

Demographic and clinical characteristics

Table 2.

Inter-observer agreement

Table 3.

Observer-electronic medical records agreement

All CERTAIN charting was completed immediately on the initial patient's assessment. Median time to complete EMR chart from the time of the patients' arrival in the ICU was 121 (interquartile range [IQR] 92–150) minutes. The poststudy user survey estimated the CERTAIN usability at 2.33 (easy). There was a subjective learning curve of 3.5 patients with no differences between learners at different stages of training.

DISCUSSION

This pilot prospective observational study evaluated reliability and validity of CERTAIN charting at the bedside during the initial resuscitation of 30 patients in the medical ICU. This study showed very good agreement between observers using CERTAIN, as well as a good agreement between CERTAIN and EMR charting, thus supporting concurrent validity and reliability of CERTAIN. CERTAIN charting was completed immediately on the end of resuscitation and 2 h earlier than EMR charting. CERTAIN observers reported good satisfaction and relatively short learning curve.

Several previous studies evaluated the validity of electronic checklists. Checklists have been associated with a reduced workload and fewer errors by ICU providers.[15] A previous study from our center addressed reliability, construct, and discriminative validity of electronic checklist during resuscitation that was performed in a simulation environment.[18] The agreement in the study, expressed as kappa coefficients, was similar to the inter-observer agreement in our study. In addition to meeting health-care quality benchmarks, checklists serve an important purpose in providing real-time guidance during tasks that require a high cognitive burden.[8,15,16] In a study from Australia, trauma resuscitation with or without the use of a checklist was compared.[19] However, in contrast to this study, the checklist user had the ability to prompt the primary team on resuscitation actions that resulted in a statistically significant increase in error-free resuscitation (16% to 21%). They also noted a decrease in morbidity, use of blood products, and incidence of aspiration pneumonia in the checklist group. The current study lacks the prompting function that is available in the full version of the CERTAIN software.[16] Since this software was developed by an international collaboration, our group sought to validate it in a step-wise fashion.[8] The reliability and validity of the content and processes of care is an essential step before intervening on decision-making. The CERTAIN investigators at our center are in the process of developing further prospective trials to validate the prompting features. Prior studies have been conducted in simulation environments, limiting the generalizability to the real world population. By the use of passive observation and recording of real patients admitted to the medical ICU, we were able to compare the reliability and applicability of the CERTAIN software to the current practice at our center.

An important finding in our study was the real-time charting function that had a significant difference of nearly 2 h as compared to the conventional EMR. The rapid increase in the use of EMRs has resulted in new challenges for patient care quality and safety.[20] Resuscitative practices nationally infrequently adopt electronic documentation tools due to the complexity of interventions, fast-paced nature and lack of dedicated resources for electronic charting.[21] In addition, the multiple data points in resuscitation practice potentially involve the use of multiple screens/portions of the EMR that might not be intuitive for practice. Using a single screen dashboard with all vital elements to resuscitation, the CERTAIN project aimed to circumvent this barrier. This study was able to demonstrate the feasibility of real-time charting during routine clinical care. The use of paper records is often complicated by missing data elements in nearly 14% of the records, which could compromise the real-time practice or post hoc analysis of patient care.[22] In addition, key data needed for registry reporting, statistical analysis, and quality improvement were contained within a paper document that required manual data extraction and transcription into other systems.[22] The employment of single-click data entry in our study potentially can serve as a solution to this issue; however, this needs validation in more dedicated studies.

The study had several limitations. First, it was performed in a closed medical ICU of one academic center that could potentially limit the generalizability of our data to other ICUs and centers. Further data from nonmedical ICUs and other multicenter studies are required. Second, due to the nature of a pilot study, our small sample size precluded any subgroup analysis. Third, usability and learning curve information was assessed based on the survey of only three observers. These were secondary outcomes for this study, and this data may provide pilot estimates for larger trials involving more observers. Furthermore, the study largely assessed objective data with a single-best response format, and hence, the applicability of our results to complex decision-making in patients with multisystem involvement needs further validation.

CONCLUSIONS

CERTAIN provides a valid and reliable way to chart resuscitation events during the 1^st h of admission to the medical ICU. CERTAIN charting is as or more complete as the EMR and is completed in real time providing the ICU team synthesized data for immediate analysis. These benefits support further evaluation of CERTAIN as a real-time charting and clinical decision support tool for acute care.

Financial support and sponsorship

Nil.

Conflicts of interest

There are no conflicts of interest.