Derivation and Validation of a Search Algorithm to Retrospectively Identify CRRT Initiation in the ECMO Patients

Summary

Background

The role of extracorporeal membrane oxygenation (ECMO) in refractory cardiorespiratory failure is gaining momentum with recent advancements in technology. However, the need for dialysis modes such as continuous renal replacement therapy (CRRT) has also increased in the management for acute kidney injury. Establishing the exact timing of CRRT initiation in these patients from the electronic medical record is vital for automated data extraction for research and quality improvement efforts.

Objectives

We aimed to derive and validate an automated Electronic Health Records (EHR) search strategy for CRRT initiation in patients receiving ECMO.

Methods

We screened 488 patients who received ECMO and a total of 213 patients underwent CRRT. We evaluated random 120 patients, 60 for derivation and 60 for validation cohorts. Following implementation of eligibility criteria, the algorithm was derived in 55 out of 120 ECMO/CRRT patients. The search algorithm was developed using first-time chart entry of ‘access pressure drop’ at CRRT initiation. The algorithm was then validated in an independent subset of 52 patients from the same time period. The overall agreement between electronic search algorithm and a comprehensive manual medical record review in the derivation and validation subsets, using ‘access pressure drop’ as the reference standard, was compared to assess CRRT initiation time.

Results

In the derivation subset (N=55), the automated electronic search strategy achieved an excellent agreement with manual search (κ =0.99, 54 were identified electronically, and 55 upon manual review). There was no time difference observed in 49/54(89%) patients, while in the remaining 5 (9%) patients time difference was within 15 minutes. In the validation cohort (N=52), agreement was 100 % (κ = 1.0, both methods identified 52 patients). Out of 52 patients, 47 (90%) had no time difference between the methods, for the remaining 5 (10%) patients, differences were within 15 minutes.

Conclusions

The use of an electronic search strategy resulted in determining an accurate CRRT initiation time among ECMO patients.

1. Background

Extracorporeal membrane oxygenation (ECMO) is a type of mechanical circulatory support system used in patients with refractory cardiorespiratory failure. Following the observation of the positive impact of ECMO in patients with severe acute respiratory distress syndrome (ARDS) due to Influenza A virus subtype (H1N1) infection in 2009, and recent advancements in technology, ECMO has been increasingly used for multiple conditions worldwide [1]. However, the benefit of its application is hampered by the high cost and potential of life-threatening complications. Bleeding, kidney injury, and infections are a few of the most feared complications observed in the practice of ECMO [2]. When acute kidney injury (AKI) patients require dialysis, it not only adversely affects short-term outcomes; but is responsible for long-term morbidity and mortality also [3]. Evaluating the exact timing of continuous renal replacement therapy (CRRT) would be a first step in looking at temporal association with ECMO initiation. It may help in exploring the role of CRRT in the prediction of the morbidity, short, and long-term outcomes of ECMO patients.

The use of an electronic health record (EHR) for patient care is advocated in accordance with The American Recovery and Reinvestment Act of 2009, and the adoption of EHR has been incentivized by US federal government [4, 5]. Within an EHR, retrospective identification of relevant clinical events including the timing of intubation, mechanical ventilation, initiation of dialysis, and ECMO requires considerable effort and resources, especially in the absence of suitably trained resources such as e.g. database analyst, medical informatics consultants or data administrators. With the widespread adoption of the use of EHR, electronic search strategies have increased in frequency [6]. Automated search strategies have shown high sensitivity and specificity in recognizing various medical entities [6, 7]. These strategies would help to accurately identify any risk factors documented in the EHR, contribute to the utilization of various quality improvement initiatives, reduce the incidence of manual errors, and hence reduce the cost and enhance the effectiveness of medical care. Therefore, electronic search strategies provide an added value to highly efficient data extraction within an electronic medical environment and allow rapid and more accurate analysis of large databases. [6, 7].

2. Objectives

There have been various automated search strategies studied in the literature; however, none has explored critically ill patients who are supported by ECMO. The aim of this study is to develop and validate an automatic electronic search strategy to identify the timing of CRRT initiation in the ECMO patients with AKI. We plan to compare the overall agreement of this strategy with a comprehensive manual medical record review, which is considered to be the reference standard.

3. Methods

3.1 Study population

The derivation and validation subsets were obtained retrospectively from the ECMO registry at Mayo Clinic in Rochester, Minnesota for patients who were admitted from January 2001, through December 2014. The total number of patients in the cohort was 213. A subset of 60 randomly selected patients was used for derivation. The search algorithm was then validated against a separate 60 randomly selected patients. After excluding patients without research authorization and with incomplete data, 55 patients in the derivation cohort and 52 patients in the validation cohort entered the final analysis.

3.2 Manual data extraction strategies (Reference Standard)

The electronic health records of the derivation and validation groups were independently reviewed by two physician researchers (PKG and TDS). Each record was evaluated for initiation of CRRT, by reviewing the critical care nephrology notes, the first appearance of ‘access pressure drop’ in the EHR, and start time of sodium phosphate and calcium chloride infusion. It was considered a necessary step to record and analyze these variables because no reference standard presently exists for CRRT initiation timings. The variables analyzed in this study were translated into time points through the use of our previously validated institutional intensive care unit (ICU) database [8]. The CRRT parameters are automatically downloaded from the ICU database, once the patient is connected to the CRRT machine. Thus, the parameters registered within the database are translated into time stamps in near real-time.

Disagreements between the two reviewers while evaluating the health records were settled by a third reviewer (RK). The research team involved in manual data extraction was blinded to the automated electronic search strategy results and vice-versa.

3.3 Automated electronic search strategy

The present retrospective study used the institutional ICU DataMart [8]. The ICU DataMart consists of patient information such as demographics, diagnoses, laboratory values, flow sheets, clinical examination, and pathologic data gathered from various electronic platforms. It allows the application of search algorithms, such as the one described here. Data is automatically captured from CRRT machine through the EHR. The ICU DataMart accuracy has been validated in separate studies [8, 9].

To develop the electronic search strategy, we first included variables such as ‘CRRT timing’ indicated in the ‘nephrology notes’ and ‘time of sodium phosphate infusion initiation’. Additional variables consisted of CRRT parameters, such as ‘access pressure time’ and ‘access pressure drop time’. The electronic search strategy was continuously refined during derivation using the following variables such as ‘catheter placement procedure note time’, ‘sodium phosphate infusion time’ and ‘calcium chloride infusion time’, or different combinations of these variables. The variable ‘access pressure drop time’ was chosen for the automated electronic search rather than the other parameters for the following reasons: 1) the access pressure drop time had nearly no missing data, because of dialysis nurses and ICU nurses’ accurate charting, and 2) access pressure drop time indicated greater agreement with higher accuracy when compared with the manual medical record review. Other parameters that were considered were not used because 1) the difference in CRRT parameters and catheter placement procedure note time varied by more than 60 minutes in 40% of the derivation subset; 2) sodium phosphate infusion starting time also lagged the CRRT initiation; in addition, sometimes it was used for correction of hypophosphatemia; and 3) calcium chloride infusion was used for both patients on CRRT and those who were not on dialysis.

We chose a difference of 15 minutes as the maximum acceptable error in order to precisely identify the start time of CRRT.

For validation of the derived search strategy, it was compared to the reference standard of comprehensive manual medical records review and an overall percent agreement plot was constructed. A κ (kappa) value was generated for both subsets. The automatic electronic search was performed independently by an anesthesia clinical research unit coordinator (MP). (A flow chart of the process is shown in ► Figure 1)

Fig. 1.

Electronic search strategy flow-diagram of patients in the ICU DataMart from January 1, 2001, through December 31, 2014.

3.4 Statistical analysis

An overall percent agreement between the electronic search algorithm and a manual medical record review was calculated. JMP statistical software (version 10.0.0; SAS Institute Inc., Cary, NC) was used for analyses.

4. Results

During the study period, 213 patients required CRRT during the ECMO run. In the derivation subset, the automated electronic search strategy achieved an excellent agreement. In comparison of the 54 patients which were identified electronically to the 55 patients with manual review, 49 (89%) had a zero time difference between the two methods, while of the remaining patients 5(9%) were within 15 minutes, and the overall agreement was found to be within 99% (κ=0.99) of the manual and electronic search strategies. The only patient in which the agreement could not be reached had a time difference of 30 minutes as the manual time entered was incorrect. This was confirmed on the review of the patient’s medical record. The mean difference in CRRT initiation was 1.06 (±0.25) minutes.

This developed search strategy was further validated in a different independent cohort of 52 patients. Of 52 patients, 47 (90%) had no observed time difference, while in 5 (10%) patients, the observed time difference was within 15 minutes. An agreement of 100 % (κ = 1.0) was found. The mean difference in CRRT initiation was 1.06 (±0.20) minutes. A plot of the overall percent agreement for the derivation and the validations subsets is shown in ► Figure 2.

Fig. 2.

Overall agreement for derivation and validation of the manual and electronic search algorithms.

5. Discussion

This study evaluating the identification of CRRT initiation time with an electronic search strategy using the institutional ICU DataMart and the manual record review achieved excellent agreement. The search algorithm was found to be both reliable and feasible. It correctly identified 90% of patients with no difference and 100% of patients within 15 minutes of CRRT starting time. Our study results were in accordance with other recently published studies highlighting the importance of electronic search strategies in data extraction [6, 7].

The development of this search algorithm is crucial in order to identify the exact timing of the initiation of CRRT. This essential first step to recording the CRRT starting time will be helpful in research and quality improvement projects such as the identification of the time to renal replacement therapy in ECMO patients, and the proportion of ICU patients developing acute kidney injury, etc. Additionally, this will provide an accurate assessment of the duration of renal replacement therapy, factors involved in AKI and its recovery, as well as any complications related to the dialysis procedure itself. However, the aim of this study was not to determine; when is the best time to initiate renal replacement therapy. This search strategy reviews large patient datasets, analyzes them, and provides an appropriate time of CRRT initiation. In turn, the method assists with the timely enrollment of patients in studies or proper and timely initiation of prophylactic and therapeutic interventions in order to improve the quality of care.

Adaptation of the EHR in clinical practice is very useful, but the amount of information gathered is enormous, which can potentially slow the research process. In order to circumvent this delay, the development of an electronic search algorithm is helpful for clinical studies, and a quality improvement standpoint. In order to recognize the factors responsible for renal replacement therapy in AKI patients supported by ECMO or other ICU patients, in general, one of the necessary first steps is to understand the timing of the initiation of CRRT. Similar algorithms can be developed for other events in patient care such as the timing of the onset of oliguria, fluid overload, hyperkalemia, and other electrolyte imbalance, or uremic encephalopathy. Our tool has the ability to enhance the quality of future research data in the field and facilitate the data abstraction.

There are several studies reporting the impact of both AKI and dialysis-requiring AKI in ECMO and general ICU patients [10, 11]. Most studies were retrospective in nature, and the numbers of enrolled patients were small [3, 11]. Obtaining the reported variables in these studies was relatively easy given the small sample sizes. However, for large retrospective or prospective studies, manual data extraction from the patient chart will be labor intensive and require significant resources. During our study the average time spent reviewing the manual records for reviewing the CRRT initiation time ranged from 5 to 15 minutes per patient, depending on the length of the ICU and hospital stay. Thus using this strategy will not only save a considerable amount of time but also make review less prone to human error as has been shown in previous studies. [6, 7, 12]

The electronic search strategy has several limitations. The main limitation of our study is its retrospective design. This algorithm will require further validation prior to use in near real time. The current algorithm was applied to the institutional database and hence external validity is needed before widespread applicability of this search strategy is adapted. However, these results can be easily reproduced by any other institution which has access to the electronic medical records with periodic data monitoring. Although this search algorithm happens to occur in a cohort of the ECMO patients, there is absolutely no reason that it can’t be applied to any other cohort of critically ill patients at any institution, who are on CRRT. As the use of EHR was mandated by the US government beginning in 2014, electronic search strategies, including ours, are likely to be adopted by other institutions with similar databases. The other potential limitation of this study is the accuracy of the DataMart in collecting vital information. Another limitation is the inherent issues associated with EMR including errors while entering the manual data. However, the DataMart’s database is regularly monitored with periodic quality checks, thus making it less prone to data recording errors. Lastly, we advise cautious interpretation of the data that is generated by applying such algorithms to big databases; as it may give false positive results, simply because enormous quantities of data can offer connections that radiate in all directions [13].

6. Conclusions

In ECMO-supported patients with AKI in ICU, CRRT initiation time can be accurately determined with high sensitivity and specificity with the help of an electronic search algorithm. This algorithm can be subsequently utilized for research, quality improvement, and clinical management purposes.