Enrollment into a time sensitive clinical study in the critical care setting: results from computerized septic shock sniffer implementation

Abstract

Objective

Recruitment of patients into time sensitive clinical trials in intensive care units (ICU) poses a significant challenge. Enrollment is limited by delayed recognition and late notification of research personnel. The objective of the present study was to evaluate the effectiveness of the implementation of electronic screening (septic shock sniffer) regarding enrollment into a time sensitive (24 h after onset) clinical study of echocardiography in severe sepsis and septic shock.

Design

We developed and tested a near-real time computerized alert system, the septic shock sniffer, based on established severe sepsis/septic shock diagnostic criteria. A sniffer scanned patients' data in the electronic medical records and notified the research coordinator on call through an institutional paging system of potentially eligible patients.

Measurement

The performance of the septic shock sniffer was assessed.

Results

The septic shock sniffer performed well with a positive predictive value of 34%. Electronic screening doubled enrollment, with 68 of 4460 ICU admissions enrolled during the 9 months after implementation versus 37 of 4149 ICU admissions before sniffer implementation (p<0.05). Efficiency was limited by study coordinator availability (not available at nights or weekends).

Conclusions

Automated electronic medical records screening improves the efficiency of enrollment and should be a routine tool for the recruitment of patients into time sensitive clinical trials in the ICU setting.

Introduction

Poor outcome in critical illness has prompted the design and conduct of novel diagnostic and therapeutic studies in the intensive care unit (ICU). Clinical trials in the ICU represent a significant investment of time and resources.¹ However, the recruitment of patients into clinical studies is a challenging task especially in the emergency and critical care setting. Inadequate recruitment can hinder ability to test hypotheses and stopping a clinical trial without reaching the planned number of patients is not uncommon.² Nearly two thirds of all clinical studies and up to 80% of trials investigating new products fail to finish on time.³ For industry that translates into a loss of $1 million a day in unrealized sales.⁴ In acute settings such as the ICU, the problem is exaggerated by a narrow time window for enrollment, as many interventions need to be applied early in the course of the disease process in order to be effective. Ethical issues and problems with informed consent present additional challenges to timely enrollment of critically ill patients.

Currently, study coordinators screen existing medical records or are notified by the bedside providers of potential patients with predetermined conditions. This approach is suboptimal, with enrollment rates in the single digits. For example, only 3% of potentially eligible patients were enrolled into an oncology trial in a community setting.⁵ In the ICU setting, enrollment rates vary from 2.6% of patients who provided consent before randomization in the PAC-Man trial (pulmonary artery catheter in the management of patients in intensive care with the participation of 65 ICU in the UK)⁶ to 10–20% of patients assessed for eligibility who were actually included in a study by Blot et al of early tracheotomy versus prolonged endotracheal intubation.⁷

Even with an emergency waiver of informed consent, conventional enrollment into a septic shock study is limited by delayed recognition and late notification of research personnel.⁸

During the past decade new technologies have been introduced to improve clinical trial recruitment in outpatient and inpatient settings.^9–12

To increase the enrollment rate for time sensitive studies in the ICU, we have recently developed and tested an automatic severe sepsis/septic shock alert (the ‘sepsis sniffer’) designed to detect these conditions with moderate sensitivity and good specificity.¹³

The objective of the present study was to evaluate the impact of the sepsis sniffer on enrollment into a time sensitive (24 h after onset) clinical study of echocardiography in severe sepsis and septic shock.

Methods

This study was conducted in the Mayo Clinic in Rochester, Minnesota, a tertiary care teaching institution with two hospitals comprising 1900 inpatient beds and 201 ICU beds. The timeline of the study and the before–after design are outlined in figure 1. After 8 months and 17 days of routine screening, we implemented the ‘sepsis sniffer’ to automatically notify the study coordinator and principal investigator via the institutional paging system of patients potentially eligible for enrollment into the clinical study of echocardiography during early severe sepsis/septic shock. System surveillance continued for 9 months and 4 days until the main study was completed.

Figure 1.

Timeline of the study.

The septic shock sniffer was tested on consecutive patients admitted to three medical, mixed, and surgical ICUs with a total of 62 beds. The characteristics of the ICUs have been previously described.¹⁴ The Institutional Review Board approved study protocols for both the clinical study and evaluation of the screening system. Informed consent was obtained for participation in the clinical study and was waived for the minimal risk screening system.

The criterion for potential eligibility for the clinical study was the presence of severe sepsis/septic shock according to standard Consensus Conference Criteria.¹⁵ In algorithmizing the criteria, we used domains representing key clinical concepts including: (1) suspicion of infection, (2) systemic inflammatory response, and (3) shock (hypotension or hypoperfusion). Because vasoactive medication often masks hypotension, the use of vasoactive medication was added to the algorithm 4 months after implementation (table 1).

Table 1.

Rules for septic shock sniffer algorithm

Variable	Rule*	Modifier (time frame)
(Domain 1) Suspicion of infection
Microbiology culture order	Blood OR stool OR lavage OR urine OR sputum	In past 72 h
(Domain 2) Systemic inflammatory response (two of the following)
Temperature	>38.6°C or <35.0°C	In past 12 h
Heart rate	>90 beats/min	Four consecutive readings
Respiratory rate	>21 breaths/min	Four consecutive readings
White cell count	>13 000 cells/ml or <4000 cells/ml	In past 12 h
(Domain 3) Hypotension or hypoperfusion (one of the following)
Systemic mean blood pressure	<70 mm Hg	Four consecutive readings
Lactate	>3 mmol/l	In past 12 h
Base	<−4 mmol/l	In past 12 h
Anion gap	>13 mmol/l	In past 12 h
Use of vasoactive medications†	Dopamine OR norepinephrine OR epinephrine OR vasopressin OR dobutamine OR milrinone	In past 12 h

^*Alert triggered by a combination of all three domains.

^†Added 4 months after sniffer implementation.

Screening criteria were optimized after initial evaluation of the algorithm.¹³ We focused on improving the sensitivity of the septic shock sniffer by accepting slightly different thresholds. Optimization was performed using MS Excel. Using iterative steps we changed specific cut-off values for the septic shock sniffer algorithm and automatically recalculated diagnostic performance. Adjusted values and rules criteria are outlined in table 1.

Septic shock sniffer

The infrastructure of the alert system is described in figure 2. An SQL-based integrative ICU database (Multidisciplinary Epidemiology and Translational Research in Intensive Care (METRIC) DataMart) gathered data within 15 min to 1 h of patient entry into the hospital electronic medical records (EMR) and served as the main data source for the septic shock sniffer.^{16 17} The sniffer engine uses the Java programming language to query METRIC DataMart for predefined rules (table 1) every hour.¹⁸ To eliminate false positive alerts for heart rate, respiratory rate, and mean arterial blood pressure, four consecutive 15 min readings are required before the trigger actions. If a patient meets the criteria of severe sepsis or septic shock (table 1), the computerized alert system sends a pager notification to the study coordinator. Once the sniffer identifies a particular patient, it does not screen that patient again for the next 14 days. A log file of page alerts is recorded in METRIC DataMart.

Figure 2.

Schematic of information flow in the notification system. METRIC, Multidisciplinary Epidemiology and Translational Research in Intensive Care; WBC, white cell count.

Clinical study using the sniffer for enrollment

The purpose of the clinical study was to comprehensively evaluate right and left ventricular performance by transthoracic echocardiography in critically ill patients with early severe sepsis and septic shock. The initial echocardiographic exam was to be performed within 24 h of severe sepsis/septic shock onset. Additional exclusions not included in the sniffer algorithm were age <18 years, pregnancy, congenital heart disease, known moderate or severe mitral regurgitation, documented coronary artery disease and cardiomyopathy.

Enrollment strategy before implementation of the septic shock sniffer

The study coordinator evaluated patients' eligibility using physicians' notes in the EMR every morning on weekdays (Monday–Friday). All patients who were in the ICU at a specific time point were evaluated. When the study coordinator found a potentially eligible patient she notified the principal investigator who performed a final assessment prior to informed consent being obtained. Sometimes, the principal investigator received independent notification of eligible patients from bedside clinicians. The study coordinator had access to all EMR data, however her evaluation was predominantly based on physicians' clinical notes stating the specific diagnosis of severe sepsis/sepsis shock, rather than on physiologic criteria, largely due to the difficulty in strictly applying physiologic criteria during medical record review.

Enrollment strategy after implementation of the septic shock sniffer

After implementation of the septic shock sniffer, the study coordinator evaluated patients' eligibility using physicians' notes in the EMR every weekday morning (Monday–Friday). In addition to the standard procedure, the study coordinator and the principal investigator also reviewed electronic alerts for eligible patients immediately they were received if this occurred during working hours. Alerts received during hours off work were evaluated on the morning of the next working day in batches before the standard review carried out.

Data collection and analysis

All alerts were recorded with the date and time and assessed for the presence of severe sepsis/septic shock by the study coordinator and confirmed by the principal investigator. The time of evaluation of the pager alert and enrollment into the study were also recorded. Patients who did not give research authorization for the use of their medical records for research purposes were excluded from the analysis.

Enrollment rates before and after the sepsis sniffer was introduced were compared using χ² statistics, using all patients admitted to the ICUs as a denominator. The two groups were compared using APACHE III score and ICU length of stay. The analysis was performed using JMP statistical software (v 6.0, SAS).

Results

The total number of admissions scanned during the study period was 8609. During the intervention, the sepsis sniffer screened 4460 ICU admissions. Out of 923 alerts sent, a trained intensivist co-investigator confirmed 317 cases of severe sepsis/septic shock (positive predictive value 34.3%). The baseline characteristics and severity of illness were similar before and after septic shock sniffer implementation (table 2).

Table 2.

Baseline characteristics of the patients

Variable	Before notification system implemented	After notification system implemented	p Value
Total number of patients with research authorization	4149	4460
Age (years), median±IQR	65 (52–76)	66 (52–76)	0.26
Sex, male (%)	48.7	51.3	0.33
APACHE III score, median±IQR	51 (37–68)	50 (36–67)	0.08
ICU length of stay, median±IQR	1.2 (0.8–2.5)	1.2 (0.8–2.3)	0.21
Mortality (%)	3.2	3.2	0.27

The average number of patients enrolled into the study doubled from four per month before pager implementation to eight per month with the paging system (68 of 4460 ICU admissions in 9 months and 4 days after pager implementation versus 37 of 4149 ICU admissions in 8 months and 17 days before implementation; p<0.05). Of 68 patients enrolled after the system was implemented, 42 were enrolled when the study coordinator was notified by the sniffer before next day morning medical record review. An additional nine patients were enrolled based on routine screening irrespective of the positive sniffer alert. Seventeen patients missed by sniffer were enrolled.

The work hours of the study coordinator (08:00–16:00 h, Monday–Friday) limited the efficiency of electronic notification. The median time to patient evaluation after pager notification was 13.7 (IQR 6–29) h. Most of the evaluations by the coordinator were carried out during the morning. Twenty nine percent of pager alerts were evaluated after 24 h, beyond the enrollment window. Twenty five percent of those (68 patients) met the eligibility criteria and could have been enrolled if evaluated by the study coordinator in a timely fashion.

In a post-hoc analysis we reviewed 17 patients who were missed by the sniffer. Table 3 provides specific reasons and potential solutions to improve sniffer performance. The number of false negatives decreased sharply after the use of vasoactive medication was added into the shock domain 4 months after sniffer implementation.

Table 3.

Review of false negatives and potential solutions

Case no.	Reason for being missed by sniffer	Explanation	Potential solution
42	RR<20, temperature normal	No SIRS criteria at the time of ICU admission; resuscitation at time of ICU admission	Check vasopressors on admission and fluids given; consider checking ED diagnoses
44	Normal BP	No shock domain criteria at the time of ICU admission because the patient was on vasopressors	Add vasopressor administration to the algorithm
47	Normal BP	No shock domain criteria at the time of ICU admission: responded well to 2 liters of fluid on arrival and quickly stabilized	Consider adding rapid fluid administration into the algorithm
49	Detected by the sniffer	Study coordinator evaluated as not having shock, but PI enrolled this patient later	Stricter gold standard criteria to reduce interobserver variability in enrollment criteria
55	Normal BP, HR and RR	Neither shock nor SIRS domain criteria at the time of ICU admission; rapid fluid resuscitation with resolution before ICU admission	Expand screening outside of the ICU; consider adding rapid fluid administration into the algorithm; no implications for clinical alerts since the condition resolved
56	Normal BP	No shock domain criteria at the time of ICU admission; was given fluids and vasopressors on admission	Add vasopressor administration to the algorithm
59	Normal HR, RR, and temperature	Neither SIRS nor shock domain criteria at the time of ICU admission; resolved in ED	Expand screening outside of the ICU for research and audit, no implications for clinical alerts since the condition resolved
60	Normal SBP	No shock domain criteria at the time of ICU admission; was given vasopressors on admission	Add vasopressor administration to the algorithm
64	Normal SBP	No shock domain criteria at the time of ICU admission; was given vasopressors on admission	Add vasopressor administration to the algorithm
65	Normal SBP	No shock domain criteria at the time of ICU admission; was given vasopressors on admission	Add vasopressor administration to the algorithm
67	Normal SBP	No shock domain criteria at the time of ICU admission: was given vasopressor	Add vasopressor administration to the algorithm
68	Normal HR, RR, and SBP	No shock domain criteria at the time of ICU admission; rapid fluid resuscitation	Expand screening outside of the ICU; consider adding rapid fluid administration into the algorithm
74	Normal HR, RR, and SBP	Resolution of SIRS and shock domain criteria with treatment prior to ICU admission	Expand screening outside of the ICU; sepsis diagnosis in clinical notes (for research and audit, no implications for clinical alerts since the condition resolved)
85	Normal RR, temperature and WBC	Did not meet at least two SIRS criteria although there was infection and shock (vasopressors)	Consider shock domain and infection domain sufficient for diagnosis even if only one SIRS criteria
86	Normal SBP	No shock domain criteria in the ICU; rapid fluid resuscitation	Consider adding rapid fluid administration into the algorithm
100	Detected by sniffer	Technically is correct, but patient was enrolled previously	Incorporate previous enrollment status into the algorithm

BP, blood pressure; ED, emergency department; HR, heart rate; ICU, intensive care unit; PI, principal investigator; RR, respiratory rate; SBP, systolic blood pressure; SIRS, systemic inflammatory response syndrome; WBC, white cell count.

Discussion

To our knowledge this is the first report concerning the use of a near-real time computer system for identification of patients with severe sepsis/septic shock for enrollment in a time sensitive clinical study.

The sniffer had moderate accuracy with a positive predictive value of 34% (one out of three pager alerts met the eligibility criteria). Despite the constraints imposed by the work hours of the study coordinator, implementation of the sepsis sniffer was associated with significantly higher patient enrollment. One of the main limitations hindering the sepsis sniffer is the imprecision of the original ACCP/SCCM Consensus Conference definition of systemic inflammatory response syndrome (SIRS).¹⁵ An international survey of physicians demonstrated that only 17% of respondents agree with the SIRS/sepsis definition and 83% agreed that a sepsis diagnosis is frequently missed.¹⁹ Our review of false negative cases (table 3) highlighted potential improvements that will be implemented in future studies. The most common cause of false negative findings was resolution of physiologic abnormalities by the time of ICU admission due to either vasopressor use or early intervention in the emergency department. Expansion of screening outside the ICU setting is important before clinical application of this tool.

The wide implementation of EMR has major potential to increase the recruitment rates of clinical trials.²⁰ However, most current EMR do not have the capability to set up and generate customized alerts. In this environment, the development of a clinical research data repository for mass screening has significant advantages. Such an infrastructure would allow the quick and convenient building of customized notification systems without negatively impacting on the production of a clinical database.¹²

Several studies explored the effectiveness of real-time notification systems. The majority of these studies are part of oncology literature where the matching of EMR data with enrollment criteria increased enrollment rates by up to 50%.^9–11 In an acute setting, Weiner et al increased notification of potentially eligible patients with sickle cell disease presenting to the emergency department from 56% to 84%.²¹ This system scanned the EMR every 2 min, but the delay before a clinical note with a chief complaint was entered into the EMR was not measured. In our study the detection of severe sepsis/septic shock syndrome was based on a combination of physiologic and laboratory data according to a standard consensus conference definition. This approach has advantages over scanning clinical notes, as note creation and transcription may take several hours.

In addition to enhancing recruitment rates, electronic alerts can decrease the workload of study coordinators by pre-selecting high risk patients. The acceptance of electronic alerts by healthcare providers for the purpose of clinical trials is high. Physicians referred more patients to clinical trials when an electronic alert system was used²² and 77% of physicians appreciated being reminded about a trial via an electronic alert and would like to see it used in the future.²³

The sepsis sniffer has high potential for future use. A recent systematic review of 27 large severe sepsis and septic shock clinical trials suggests that future studies should be restricted to a 24 h time window from onset of organ dysfunction or shock and the recruitment period should not exceed 24–30 months.²⁴ Identification of patients with severe sepsis should increase the number of patients enrolled in clinical trials and also help the reporting needs of internal and external agencies (currently it is difficult to report compliance with evidence based interventions for severe sepsis as we fail to capture many patients with severe sepsis).

The before–after design and the absence of concurrent controls is a major limitation of our study. During the intervention period the majority of patients were enrolled after the study coordinator was prompted by the sniffer, but we do not know how many of these patients would have been picked up anyway.

Alternative explanations for the increased enrollment include improved study coordinator skills, notification by bedside providers, and regression to the mean. Our alert system was designed to improve poor enrollment and to supplement rather than replace routine practice. While we did not formally measure the time spent by the study coordinator before and after alert implementation, the perceived efficacy of the new system in both decreasing coordinator workload and increasing enrollment during the narrow time window precluded a parallel design. We did not measure the exact number of notifications by bedside providers before and after sniffer implementation, however the study coordinator did not report any changes over time. The majority of cases were picked by the co-investigator (JP) during his clinical work. The relatively modest positive predictive value (34%) is a reasonably good trade off, given that timeliness and sensitivity took priority over specificity, and the false positive cases could be filtered out by the study coordinator. Future design improvements include but are not limited to: taking additional variables into account (eg, adding serum lipase to exclude pancreatitis which frequently mimics sepsis), trend analysis of serial vital signs, the addition of natural language processing of nursing and physician notes, and specific data mining techniques.

The major limitation affecting the efficiency of the sepsis sniffer was the current working hours of the study coordinators who evaluated alerts only during the daytime. Moreover, most of the alerts were evaluated in batches the following morning outside the time window for participation in a clinical trial.

The slightly longer observation period after sniffer implementation was unlikely to have influenced the observed results as more patients were under observation during the period before sniffer implementation than after (4460 before versus 4149 after).

An additional limitation is related to possible diagnostic bias in the ordering some of the tests. While vital signs, white blood cell count and electrolytes are routinely investigated, other tests such as blood cultures and lactate are provider dependent. Our ICU data mart obtains data in near-real-time rather than real-time and future applications will need to be built within the EMR itself to allow for instantaneous alerting. Finally, our results may not be generalizable since they are based on data from a single institution.

Clinical research in critical care settings is particularly difficult due to the complex nature of physiologic syndromes such as sepsis and the importance of timely investigation (in hours rather than days). Conventional chart review by study coordinators is suboptimal and even non-perfect informatics tools can greatly enhance traditionally poor enrollment. Complex physiologic and laboratory criteria are very difficult (and time consuming) to apply appropriately during chart review, and study coordinators frequently rely on the physician interpretation in the clinical notes. However, the notes are often inconsistent and rarely timely.

Our further work with the sepsis sniffer is focusing on improving task specific performance using different definitions for practice (high specificity) and enrollment for research studies (high sensitivity).

Conclusions

The complexities of the ICU environment and the frequently narrow enrollment window pose particular challenges to recruitment into clinical studies involving acutely ill subjects.

Although limited by the availability of study coordinators, an automatic severe sepsis/septic shock sniffer tool was associated with markedly increased enrollment into a time sensitive clinical research study. Advantages also included a decrease in both screening time and overall time to study completion. Automated EMR screening will likely become a routine tool for the recruitment of patients into time sensitive clinical trials in the ICU setting.