Detecting organisational innovations leading to improved ICU outcomes: a protocol for a double-blinded national positive deviance study of critical care delivery

Abstract

Introduction

There is substantial variability in intensive care unit (ICU) utilisation and quality of care. However, the factors that drive this variation are poorly understood. This study uses a novel adaptation of positive deviance approach—a methodology used in public health that assumes solutions to challenges already exist within the system to detect innovations that are likely to improve intensive care.

Methods and analysis

We used the Philips eICU Research Institute database, containing 3.3 million patient records from over 50 health systems across the USA. Acute Physiology and Chronic Health Evaluation IVa scores were used to identify the study cohort, which included ICU patients whose outcomes were felt to be most sensitive to organisational innovations. The primary outcomes included mortality and length of stay. Outcome measurements were directly standardised, and bootstrapped CIs were calculated with adjustment for false discovery rate. Using purposive sampling, we then generated a blinded list of five positive outliers and five negative comparators.

Using rapid qualitative inquiry (RQI), blinded interdisciplinary site visit teams will conduct interviews and observations using a team ethnography approach. After data collection is completed, the data will be unblinded and analysed using a cross-case method to identify themes, patterns and innovations using a constant comparative grounded theory approach. This process detects the innovations in intensive care and supports an evaluation of how positive deviance and RQI methods can be adapted to healthcare.

Ethics and dissemination

The study protocol was approved by the Stanford University Institutional Review Board (reference: 39509). We plan on publishing study findings and methodological guidance in peer-reviewed academic journals, white papers and presentations at conferences.

Strengths and limitations of this study.

• This study is a methodologically innovative translation of the positive deviance approach to health services research and incorporates both qualitative and quantitative rigour.

• A national database of intensive care unit (ICU) care in the USA is used, and the methodology is a mixed-methods approach that triangulates between qualitative and quantitative data sources.

• The project is likely to inform the organisation of care delivery in ICUs and both positive deviance and rapid qualitative methodologies in healthcare.

• The database is limited to ICUs with telemedicine capabilities, and case-mix adjustment using Acute Physiology and Chronic Health Evaluation IVa is imperfect. Study participants and site visitors may also not be able to accurately ascertain which innovations drive performance.

Introduction

Critical illness represents an enormous burden in the USA, with more than 5 million patients admitted annually to intensive care units (ICUs).¹ Caring for these patients consumes a disproportionate amount of resources; despite comprising fewer than 10% of all hospital beds, ICUs account for 13.4% of total hospital costs and 0.66% of the national gross domestic product.² This burden will likely increase with the ageing population, as both utilisation rate and the proportion of beds allocated to intensive care increase.^2–5

Yet, the quality of care delivered varies dramatically between units and hospitals. ICUs differ widely in their rates of compliance with best practices and rates of avoidable complications (eg, hospital-acquired infections).⁶ Risk-adjusted mortality also differs among ICUs, with studies suggesting that high-performing ICUs in the country have up to 10–12 fewer deaths for every 100 patients than the lowest performing ICUs, even after controlling for factors like discharge practices and patient demographics.⁷ These trends have been confirmed in more recent studies of ventilated patients.^{8 9}

Variations in performance are likely driven by differences in ICU organisation and practices, rather than by access to technology.^{10 11} Modern ICUs are more an organisational innovation than a technological one, matching a concentration of personnel and resources for any type of critically ill patient. Previous research has identified the association between organisational factors like nurse-to-patient ratios, daily care plans and usage of care bundles and  improved risk-adjusted mortality.^12–14

Unfortunately, innovations in organisation and practice are not well described in the critical care literature. Hospitals do not typically share their innovative practices with one another, and data to compare ICU performance are not readily available. Although some practices may be published, context is frequently not reported in sufficient detail to ensure successful implementation.¹⁵ All these factors obscure our ability to identify which aspects of critical care organisation and practices help drive performance.

Positive deviance is one methodology that may offer additional insights. This approach assumes that innovations that address problems common to many organisations have already been developed and can be detected by studying positive outliers before being tested and disseminated.^{16 17} Originating from global health, the approach has been used successfully in a wide variety of settings to improve healthcare quality, including diabetes management in primary care practices and hospital door-to-balloon times in response to acute myocardial infarction.^{18 19}

However, a systematic review of positive deviance studies in healthcare found research quality to be low and there have been very few applications of the approach in the critical care setting.¹⁶ Highlighting the need for increased rigour, a previous study, which used qualitative site visits, failed to identify the differences between ICUs associated with performance.¹¹ The goal of this research protocol is to describe our methods for conducting a positive deviance study in critical care. Specifically, we sought to identify organisational innovations in the delivery of critical care, adapting the first two steps of the positive deviance approach to generate hypotheses as to which innovations explain variation in ICU utilisation and quality of care. A secondary objective was to identify potential organisational structures, processes and contexts that may explain this variation. Through these aims, we hope to detect innovations in intensive care and support an evaluation of how positive deviance and rapid qualitative inquiry (RQI) methods can be adapted to healthcare.

Methods and analysis

Conducting a positive deviance study requires four steps: (1) identify outliers within an area of interest, (2) use qualitative approaches to generate hypotheses to explain their performance, (3) test hypotheses in a larger sample and (4) disseminate evidence about best practices.^{20 21} Our strategy uses a blinded, retrospective approach in the two first steps. We analysed a national database of ICUs to develop a study cohort of five positive outliers and five comparator ICUs. This quantitative phase will be followed by in-depth qualitative work at these ten sites, where we will build comparative case studies on their innovations and themes.

Quantitative phase: identifying outliers

Data source

We used data from the Philips eICU Research Institute (eRI) database, containing over 3.3 million patient records from over 50 health systems from 2003 to 2015. All ICUs in the database have implemented the Philips eICU telemedicine system. Data for ICU admissions include vital sign measurements, quality metrics, medication orders and patient laboratory values. All the ICUs in the study were given an opportunity to opt out of the study, and the protected health information of individual patients was not included. The database includes data from over 400 hospitals; as of 2014, there were 5686 acute care hospitals in the USA, all of which had at least one ICU.²²

ICU cohort selection

Inclusion criteria included all hospital units that contributed data to the Philips eRI database between 2013 and 2015. We excluded the hospital units that did not participate for all 3 years and self-identified step-down or intermediate care units. To minimise variation from small sample sizes, we also excluded low-volume ICUs, defined as ICUs with fewer than 300 discharges per year. The final cohort included 276 ICUs that cared for a total of 3 70 278 patients over 3 years. These ICUs form a geographically diverse sample of ICUs with eICU capabilities.

Outcome measurements

Primary outcomes included mortality and length of stay for patients admitted to the ICU, since these parameters reflect both ICU quality and utilisation. While mortality rates are generally low in critical care and thus insensitive to use in comparisons,^23–25 rates of deaths are sufficiently high enough among ICU patients to be used as a quality indicator.⁷ As patients may be transferred elsewhere in the hospital as death nears,²⁶ ICU patient mortality rates were calculated using deaths that occurred both in the ICU (in-ICU mortality) and after transfer elsewhere within the hospital (combined post-transfer mortality).

The eRI database does not include any cost estimates. We used length of stay used as a proxy for resource utilisation, since up to 85% of ICU costs are explained by length of stay alone.²⁷ In this study, we calculated a mean residual for each ICU, using the difference between observed and expected lengths of stay, as predicted by the Acute Physiology and Chronic Health Evaluation (APACHE) IVa algorithm. Patients who died before discharge were excluded. As with mortality, we calculated length of stay including only ICU lengths of stay (in-ICU length of stay) and including days after transfer elsewhere within the hospital (combined post-transfer length of stay).

Patient cohort selection

The variation in outcomes between ICUs is mostly dominated by those who are very healthy or very sick. For example, ICU metrics are greatly skewed by low-risk patients admitted to the ICU purely for monitoring purposes and by high-risk patients for whom death may be a likely outcome. Consequently, only patients who have a predicted risk of death between 2% and 20% were included, as predicted by the APACHE IVa algorithms.²⁸ Patients without calculated APACHE IVa scores were excluded. These limits were based on expert consensus among clinicians who are familiar with the APACHE IVa scoring system.

Patients transferred between hospitals were also excluded from the study. Transfer status from another institution is an independent risk factor for mortality, even after controlling for case-mix.²⁹ Small numbers of patients transferred dramatically affect mortality rates,^30–32 and transfers are excluded from the APACHE IVa models.³¹ In order to control for this ‘transfer bias’, we excluded all patients who were transferred from another institution. We also excluded patients with extreme outlier unit lengths of stay more than  300 days.

Direct risk standardisation

In order to enable direct comparison of outcomes between each ICU, direct risk standardisation was used to adjust for variations in case-mix.³³ In summary, we calculated a weighted average for each outcome variable using two percentage point increment risk groups based on APACHE IVa-predicted ICU mortality (eg, 2%–4%,… and 18%–20%). The weights were equal to the proportion of the number of patient records within each risk group. Weighted average mortality rates and lengths of stay were calculated for all patient records for each individual ICU. ICUs with less than 300 patient records for those within the 2%–20% APACHE IVa-predicted mortality were excluded to eliminate extreme variations due to small sample sizes.

Bootstrapped variance and percentile CIs

As risk adjustment was performed using direct risk standardisation, all adjusted outcome variables were weighted means. Unlike the arithmetic mean, no analytical analogue of the SE exists for weighted means.³⁴ Therefore, we estimated CIs through bootstrapping.^{35 36} All outcome variables were calculated for each ICU, using 5000 resamples with replacement equal to the total number of patient records for each individual ICU. Variance and percentile CIs were then calculated for each ICU.

Outlier identification and false discovery rate control

Outlier and comparator ICUs were defined as ICUs with CIs that do not overlap with the population mean (α≤0.05). p values were generated for each ICU using a two-sided Student’s t-test and then adjusted for false discovery rate (d<0.05) using the Benjamin-Hochberg procedure.³⁷ This process was repeated for each of the four outcome variables (ie, in-ICU mortality, combined post-transfer mortality, in-ICU length of stay and combined post-transfer length of stay) and visualised using caterpillar plots sorted by CI limits. ICUs identified as outliers on all four outcome variables were placed into respective positive outlier and negative comparator groups.

Qualitative phase: detecting innovations

Site selection

Two members of our study team (HC and MR) were provided with an unblinded list of ICUs identified as positive outliers and negative comparators. A purposive sample of five positive outliers and five negative comparators were selected using a maximum variation approach based on the following institutional characteristics: (1) ICU type, (2) patient volume, (3) academic affiliation, (4) presence of intermediate care units, (5) case-mix of ICU, (6) geographic locale, (7) urban or rural and (8) health system.¹⁸ The site visit teams were then provided with a blinded list of these ICUs for recruitment. The sample size of 10 sites is based on previous research establishing 10 sites as likely to achieve thematic saturation for positive deviance studies in healthcare.²¹

Site visits

We adapted the team-based RQI methodologies used in public health and applied anthropology, which rests on building rapport quickly, triangulating across multiple sources of data and a multidisciplinary research team.³⁸ The blinded RQI team includes a surgeon and systems engineer (JKJ), a registered ICU nurse and administrative fellow (DB) and a healthcare researcher (RP)—all trained by two applied anthropologists (HC and HK). The research team will collect and analyse three key data sources: (1) semistructured interviews, (2) unstructured observations and (3) extant data.

The Consolidated Framework for Implementation Research (CFIR) will be used as a theoretical framework to guide both data collection and subsequent analysis.³⁹ CFIR is a determinant framework consisting of constructs known to be associated with effective implementation and intended to guide evaluations and implementation strategy.^{39 40} As the CFIR constructs include interventions, individuals, organisational context and organisational processes, this framework provides both a typology and a terminology to evaluate interventions and their context.

Semistructured team interviews and focus groups

Bedside staff and unit managers will be recruited for interviews using a combination of key informant, snowball and opportunistic sampling.⁴¹ Recruitment will occur using a maximum variation approach, aiming to capture a wide variety of perspectives at each site from across the hierarchy, including doctors, nurses, nursing technicians and unit managers. Teams will recruit at least six to eight participants at each site, a sample size found previously to be usually sufficient for thematic saturation in healthcare positive deviance.^{21 42}

All interviews will be semistructured and use an interview guide that broadly addresses three key domains: unit practices and communication, quality improvement and relationships between management and frontline staff (see online Supplementary material). The interviews will seek to identify innovations in these key domains and generate testable hypotheses that may explain the variations in performance.⁴³ All interviews will be conducted in private settings, digitally recorded and transcribed verbatim by professional transcriptionists.

Unstructured observations

Observational data are particularly important for rapid qualitative approaches, as they provide a point of triangulation against data from interviews.⁴⁴ Our strategy requires observational data obtained using ethnographic methodologies, which are designed to access the typical routines and conditions of a field site.⁴⁵ Site visit teams will conduct at least 2 hours of direct observation in each ICU, including physician rounds, nursing shift changes, cardiopulmonary resuscitations and fixed observation at nursing stations and eICU command centres. Each researcher will systematically generate descriptive field notes, including observed behaviours, processes and environmental features.⁴¹ These unstructured observations also provide opportunities to build rapport and conduct informal interviews with bedside staff.

Extant data

Collection of contextual data is a critical component of RQI and provides an additional basis from which hypotheses can be triangulated.³⁸ For example, site visit teams may encounter training documents, written policies, news reports or locally collected data. With permission, these data will be digitised into the research database and analysed as described below.

Rapid continuous constant comparative analysis

This project will adapt a team-based, continuous analysis methodology commonly used in rapid qualitative approaches.³⁸ Considered critical to a team ethnographic approach, site visit teams will debrief as often as possible, reviewing field notes and interviews to generate potential hypotheses and innovations for each field site. The main purpose is to generate analytical field notes in a modified grounded theory approach, generating themes and causal explanations grounded in the data.^{46 47} While classic grounded theory emphasises a primarily inductive approach, we will include a mixed grounded theory and content analysis as typical of rapid qualitative research.^{38 47}

All field notes, preliminary reports, interview transcripts and any extant data are then imported to Dedoose, a qualitative analysis software designed for teams.⁴⁸ All data will then be inductively coded using a combination of grounded theory and constant comparative methods, extending the formal codebook of themes identified during team debriefs. As site visit teams remain blinded to each site’s outlier status, a constant comparative method will be used to generate causal models of factors and innovations, assessing the possibility that a field site is a positive outlier or negative comparator site in turn. Additional field notes are generated in this process (‘memoing’), and a preliminary report for each site visit is generated.³⁸

Cross-case analysis

All members of the study team will then be unblinded as to each sites’ outlier status, and all data sources will be analysed using a cross-case method.⁴⁹ Relevant qualitative and quantitative data points will be entered into a matrix, organised by themes of interest identified during site visits and the outlier status of each site. The data will then be interrogated for patterns, themes, similarities and differences between the outlier and comparison sites. Causal models developed during the generation of preliminary reports will then be extended across multiple sites.

Ethics and dissemination

The study protocol was reviewed and approved by the Stanford University Institutional Review Board (reference: 39509). Verbal informed consent will be obtained from all participants, and interviews will remain confidential and de-identified. Any study findings will only be reported in the aggregate, and individual ICUs will never be identified in publications. Participating ICUs will not be disclosed their outlier status, but all publications and reports will be shared with the recruited sites. Potential innovations will also be disseminated to participants and nationally through the work at the Clinical Excellence Research Center at the Stanford University. We plan on publishing study findings and methodological guidance in peer-reviewed academic journals, white papers and presentations at academic medical conferences.

Study status

The quantitative portion of this study is complete. Qualitative data collection began in September 2016 and  completed in April 2017. Qualitative data analysis will be completed by September 2017.

Discussion and limitations

We aim to extend positive deviance methods into a national study of intensive care. By focusing only on a subset of patients who are most likely to have lengths of stay and mortality rates affected by organisational processes and practices, this study aims to detect new innovations in the delivery of critical care. These innovations can then be tested in subsequent studies and disseminated broadly if found to be efficacious.

There are several limitations to this study. First, the database includes only ICUs that subscribed to the Philips eICU programme. However, as the main objective of this study is to identify new organisational innovations that may drive ICU performance, the fact that all ICUs in this study have telemedicine capabilities ensures a similar level of technological access. In the USA, there are few other national databases of ICU quality, and the database is likely one of the most comprehensive data sources available.

Second, APACHE IVa is an imperfect measure of disease severity, although it remains one of the most widely used and best validated measures.^{50 51} As the main purpose of the quantitative portion of this project is to identify outlier ICUs, likely to harbour organisational innovations, we believe the large sample sizes in this project will also protect against this limitation.

Finally, an intrinsic risk of the positive deviance approach is that success is dependent on the ability of either the researchers or the study participants to identify the innovations, leading to variations in outcome. Although the double-blinded nature of this study maximises our ability to correctly identify successful innovations, there is an unavoidable risk that no new innovations will be identified. Replication of positive deviance studies can also be challenging, as differing site visit teams may identify different innovations as worthwhile.

This protocol, however, also contains several novel features to further the translation of positive deviance methods to healthcare services research. First, we are conducting the study with both qualitative and quantitative rigour, responding to previous criticisms of the method. Second, this study is the first to use a double-blinded strategy, as both study participants and site visit teams are not disclosed the outlier status of individual ICUs. These methodological innovations will allow us to evaluate the usage of positive deviance and RQI methods in healthcare and test rapid team ethnography as a research tool. Our hope is that these methodological innovations will make a significant impact in improving healthcare delivery and outcomes for critically ill patients.