Survival after Acute Myocardial Infarction (SAMI) Study: The Design and Implementation of a Positive Deviance Study

Abstract

Positive deviance studies combining qualitative and quantitative designs – a mixed methods approach – can discover strategies to produce exemplary performance. We present the Survival after Acute Myocardial Infarction (SAMI) study, a national positive deviance study to discover hospital strategies associated with lower 30-day hospital risk-standardized mortality rates (RSMRs). There is marked variation across hospitals in 30-day hospital RSMRs for patients with acute myocardial infarction (AMI) and little information about what accounts for differences in performance. We first conducted a qualitative study of hospitals in the U.S. (n=11; 158 key staff) that ranked in the top 5% of RSMRs for each of the 2 most recent years of data (2005–2006, 2006–2007) from the Centers for Medicare & Medicaid Services (CMS) at the time of sample selection and in the bottom 5% for contrast, with diversity among hospitals in key characteristics. Using hypotheses generated in this qualitative stage, we constructed a quantitative survey that was administered in a cross-sectional study of acute care hospitals in the U.S. operating from July 1, 2005 through June 30, 2008 that publicly reported CMS data for RSMRs during this time. We included hospitals with at least 75 AMI discharges during the 3-year period. Of the 600 hospitals we attempted to contact, 10 had closed, leaving a final sample of 590, of which 537 responded (91%). This type of study, using a positive deviance approach and mixed methods design, can generate and test hypotheses about factors most strongly associated with exemplary performance based on practices currently in use.

Introduction

The study of positive deviance in health care, which focuses on identifying best clinical practices of exceptionally high-performing organizations, holds promise for improving the care and outcomes of patients. Organizations with exemplary performance have discovered ways to produce uncommon outcomes. A positive deviance approach takes advantage of this natural variation in performance through intensive study of top performers, or ‘positive deviants,’ to develop a best practices evidence base that, because it originates from peer organizations, is robust, credible and more likely to be adopted and sustained by others.(1) A positive deviance study can benefit from a mixed methods design,(2–4) which combines qualitative and quantitative studies in a single program of inquiry. In a positive deviance study, the qualitative phase occurs first, producing insights regarding aspects of the organization that cannot be measured quantitatively,(5–6) and generates hypotheses about the factors responsible for exemplary performance. These hypotheses are tested in a subsequent phase using quantitative approaches in a large, representative sample of organizations.

We present the Survival after Acute Myocardial Infarction (SAMI) study, a national mixed methods study to discover hospital strategies that are associated with higher survival rates for patients hospitalized with an acute myocardial infarction (AMI). We first convey key principles and considerations in designing and implementing positive deviance studies.

Conceptualizing a Positive Deviance Study

Origins in Biology

Our early conceptualization of studying positive deviance in organizational performance derived from observing the utility of extreme phenotypes in genetic studies. An individual with an extreme phenotype represents unusual characteristics that may reveal insights about underlying biological mechanisms of health and disease. For example, Richard Lifton, a prominent geneticist at Yale, has developed studies of individuals and families with extreme phenotypes to identify gene mutations that play important roles in specific conditions and diseases.(7)

Positive Deviance in Organizations

We believe that the success in learning about outliers in biology has relevance to understanding factors that favor success or failure in organizations. Until recently, clinical performance has not been consistently measured; we may thus expect great variation in practice among health care organizations. With few negative ramifications associated with poor performance, there has been no competitive advantage for organizations to excel in this area.

The premise of the positive deviance approach in organizations is that solutions to organizational problems can be found among organizations that excel. These positive outliers have developed strategies that enable exceptional performance with capabilities and resources that exist among the community of organizations. The hope is that these strategies can be generalized, adopted and implemented by the broader community to produce similar results. These concepts also have precedents in public health studies.(8–10)

Prior Applications

In cardiovascular outcomes research, we have pursued positive deviance studies with mixed methods designs to address suboptimal prescription of beta-blocker medications at hospital discharge for patients hospitalized with an AMI and for long delays in providing primary percutaneous coronary intervention for patients with an ST-segment elevation myocardial infarction (door-to-balloon time).(11–15) In each case, we conducted qualitative studies of top performers and then tested the hypotheses in a national survey of institutions, correlating performance with organizational strategies. These studies resulted in knowledge that was integrated into quality improvement efforts. The door-to-balloon project led to a national campaign to facilitate adoption of the success strategies and contributed to marked improvements in national performance.(16–18)

Guiding Framework

Consistent with health care organizational theory, we view quality as primarily determined by organizational context, with particular attention to the complexity of how organizational features interact to influence outcomes.(19–20) We also recognize that most care strategies are complex interventions.(21–22) Accordingly, the positive deviance approach characterizes not just which processes and practices are present in top-performing organizations but also the environment (e.g., organizational culture, leadership support, norms of behavior) in which they are implemented.

Conditions Necessary for a Positive Deviance Study

As has been previously described,(1)several conditions are necessary in order to conduct a positive deviance study. In health care organizations, the approach requires widely accepted and valid performance measures for organizations. For instance, in the case of hospital care there are several specific, validated and publicly-reported performance measures; therefore, hospitals can be ranked according to performance and positive deviants within the industry can be identified. The positive deviance approach also requires that there be variation in organizational performance, with some organizations achieving exceptional performance. In addition, the approach is effective when hypotheses generated from the experience of top-performing organizations can be tested in a large, representative sample of organizations. Last, the perception among potential adopting organizations that improvement in the selected performance measure is important for their organizations’ success is useful for effective dissemination.

Implementation of a Positive Deviance Study

Implementing a positive deviance study for improvement of quality of health care includes the following four stages previously identified.(1)

Stage 1: Identify ‘Positive Deviants’

Positive deviants are identified using widely endorsed and accessible performance measures for organizations. For example, in the case of hospital care, there are specific, validated and publicly-reported performance measures;(23–28) therefore, hospitals can be ranked and positive deviants within the industry can be identified.

Stage 2: Generate Hypotheses

The qualitative component characterizes fully the factors that contribute to the clinical performance of the organization. This stage, which can be time consuming and resource intensive, provides information for generating hypotheses about what accounts for exceptional performance and is key to the construction of sound quantitative measures to be used in the survey phase. The sampling and analysis for qualitative research are distinct from those for quantitative studies.(3, 29–31) Sites are selected purposefully; qualitative research requires a diverse rather than representative sample with adequate range across variables that might influence the selected phenomenon. Importantly, not all findings from the qualitative component can be translated into a reliable quantitative measure; thus, the findings from the components are complementary.

In contrast to quantitative studies, in which analyses are conducted only when data collection is complete, qualitative data collection and analysis occur iteratively, commonly using the constant comparative method.(32–34) Although the broad research question is guided by extant literature and our prior work, we employ the approach described by Lofland and others (35) in which the data collection activity is deliberately open ended. The interview guide comprises open-ended questions and probes to be used at the interviewer’s discretion in order to elicit additional detail or provide clarification of a concept.(36) The respondent directs the course of discussion as much as possible and interviews are interactive. These methods require highly experienced interviewers with the ability to establish rapport with respondents, use discussion guides flexibly, and use probes and follow-up questions to draw out responses. Interviewers must be skilled in passive listening and the use of neutral, non-judgmental language in encouraging respondents to provide detail. Simultaneously, interviewers must maintain control of the data gathering through vigilant attention to the purpose of the interview, asking the right questions, and giving appropriate verbal and non-verbal feedback.(3, 34)Although the interview guide may evolve as data collection continues, our approach is to construct a sufficiently open guide such that the questions remain fairly constant over the course of the study.

Stage 3: Test Hypotheses

The quantitative component of the study tests the hypotheses by analyzing whether and how specific strategies and organizational characteristics are statistically associated with clinical performance. These hypotheses are drawn from qualitative research and feedback from individuals involved in providing the clinical care.

The survey approach can identify strategies most strongly associated with performance, and because the sample is randomly selected and representative, findings can be reliably generalized to the larger universe from which the sample was drawn. The approach is better at identifying strategies than proving that strategies are not useful because, in some cases, useful strategies may be implemented poorly at many institutions, diluting the average effect of an approach. The quantitative phase does not provide a means for testing the benefit of strategies that are implemented by very few institutions.

When the success factors are identified, there is a choice about whether to promote adoption or to test these strategies in additional studies such as a randomized trial. If the findings are sensible and identify a set of inexpensive and practical strategies, and if the problem is large and in need of attention, it is reasonable to move to the dissemination phases. However, if the strategies identified are controversial, disruptive or expensive and the problem does not require immediate intervention, it may be advisable to pursue another study to determine whether the identified strategies can be refined and further tested in a controlled setting. This approach assumes that potentially confounding factors can be successfully controlled and randomization is feasible and ethical.

Stage 4: Disseminate Findings

In the final stage, the research team works with key stakeholders, including potential adopters, to disseminate the information and enhance adoption. Factors that are important to successful adoption include the credibility and perceived simplicity of the recommendations, alignment with hospitals’ strategic goals, practical implementation tools, and breadth of the network of peer hospitals.(37–38) Involving potential adopters in the development and testing of proven strategies can promote the pace and scope of uptake.(38–39)

SAMI: A Positive Deviance Study

We present the ongoing SAMI study, which reflects the core principles and practices of a positive deviance approach. SAMI expands the use of positive deviance beyond the prior studies of process and intermediate outcomes, to the study of excellence in patient outcomes. In the following sections we provide an overview of the rationale for the study, define the aims, and describe the general implementation steps for each aim.

Rationale

Hospitals vary substantially in their 30-day RSMRs.(40–41) Despite increasing interest and incentives to reduce mortality rates for patients with AMI, few studies have identified organizational factors associated with exemplary performance. Variation in hospital characteristics of the core process measures explains little about the factors that differentiate hospitals by their short-term outcomes.(42–43) Such variation in performance is a necessary condition for a positive deviance study.

An impediment to research on this topic has been the absence of robust risk-adjusted mortality models to account properly for case-mix and volume differences among hospitals. The ability to measure the hospital phenotype, as noted, is a critical condition to these types of studies. Recently, however, this barrier has been overcome. We have derived a claims-based hierarchical risk-adjustment model (26) in which hospital-level RSMRs compare well with those of a medical record data-based model. Our model was published in the peer-reviewed literature and endorsed by the National Quality Forum, and is used by the Centers for Medicare & Medicaid Services (CMS) for public reporting.

Specific Aims

The SAMI study has the following specific aims:

1. To generate hypotheses concerning hospital strategies that may be associated with hospital RSMRs for patients with AMI;

2. To test the hypotheses developed in Aim 1 and determine hospital efforts that are associated with hospital RSMRs for patients with AMI.

Research Methods for Aim 1

To develop the sample for Aim 1, we evaluated hospitals according to their RSMR reported by CMS’ Hospital Compare (http://www.hospitalcompare.hhs.gov/) in 2005–2006 and in 2006–2007. We selected hospitals that ranked in the top and bottom 5% of performance during both years and were diverse in areas such as AMI volume, teaching status and socioeconomic status of patients. The aim was to identify “information-rich” participants that have certain characteristics, detailed knowledge or direct experience relevant to the phenomenon of interest, as recommended by experts in qualitative research.(44) We employed established methods for developing samples for qualitative and mixed methods studies.(29–30, 34) We excluded hospitals that did not perform primary percutaneous coronary intervention in order to target those with sufficient experience with ST-segment elevation myocardial infarction. We continued with the hospital visits until we reached theoretical saturation, i.e., the point at which no new concepts emerge from reviewing successive data from a sample that is diverse in pertinent characteristics and experiences.(33, 45)

We analyzed data from interviews with 158 staff who were most closely involved with AMI care at 11 hospitals including 7 in the top group and 4 in the bottom group. The qualitative research team had members with expertise in cardiology, emergency medicine, health services research, quality improvement, nursing, organizational psychology, and social work; all had experience in conducting in-depth interviews. Each site visit included 3–4 interviewers. The interviews were typically 1 hour in duration and were audio-taped and professionally transcribed.

An organizational psychologist conducted formal debriefing sessions with each site visit team promptly upon return to inform subsequent data analyses.(34, 46); reflections were captured in session notes kept by the organizational psychologist. These reflections included observations made by team members during the site visit; unanticipated or seemingly inconsistent data; broad reflections of the experience; and perceptions of hospital organizational dynamics. These notes were shared with the full team and informed the analysis of transcribed interviews both within and across hospitals.

A 6-member multidisciplinary team performed data analyses using the constant comparative method,(30, 33–34) in which essential concepts from interview data were coded to extract recurrent themes across the data. Those who attended specific site visits reviewed the coded transcripts from those visits.

We conducted interviews between December 2008 and December 2009. The interviews were conducted after the period in which performance was assessed because of delays in the availability of Medicare data; however, respondents were asked to describe changes in AMI care that had occurred over the past several years.

Initial Results

Six domains characterized participants’ experiences in hospitals with high and low performance in RSMRs: 1) hospital protocols and processes for AMI care; 2) organizational values and goals; 3) senior management involvement; 4) broad staff presence and expertise in AMI care; 5) communication and coordination among groups; and 6) problem solving and learning.(47) The high- and low-performing hospitals differed in 5 of the 6 domains; only for hospital protocols and processes for AMI care were no important differences found.

Research Methods for Aim 2

To accomplish the second aim, we selected all Medicare-certified hospitals that admitted at least 75 Medicare fee-for-service beneficiaries with a principal discharge diagnosis of AMI from July 1, 2005 through June 30, 2008. From these, we selected a random sample to be contacted for the Web-based survey. We are matching their responses with available risk-adjusted mortality data from a similar period.

We employed a Web-based survey (Appendix) with telephone follow-up with quality improvement or cardiology directors or their designees. We sent a letter of invitation to participate in the study to the chief executive officers of all selected hospitals and asked them to identify the person most involved in AMI quality improvement efforts, typically the director of quality improvement or director of cardiology, who we then contacted to complete the survey. Respondents were instructed to coordinate with other relevant staff to complete a single survey reflecting the hospital’s strategies.

We used a comprehensive, multistage procedure for survey development, using state-of-the-art methods.(48) First, we made use of the qualitative study to identify de novo factors potentially associated with RSMRs from the perspective of those most closely involved with AMI care. Second, we completed cognitive interviews (n=8) with staff who had roles similar to those of potential survey respondents (e.g., directors of quality assurance) to assess saliency of constructs, language and terminology, and adequacy of response options. The Web survey required 20–30 minutes to complete. It was sent to 590 randomly selected hospitals, with a response rate of 91% (n=537 hospitals).

Several strategies were implemented to enhance the reliability and validity of data received via Web survey and telephone follow-up. As noted above, the survey instrument underwent substantial pre-testing. In addition, research staff who had contact with participating hospitals were blinded to mortality rates of hospitals and hypotheses of the study. Last, interviewers underwent training regarding email contact with respondents and telephone follow-up techniques so as to not influence responses.

This work was supported by grant R01 HS16929 from the Agency for Healthcare Research and Quality and United Health, and grant 20100501 from The Commonwealth Fund. The authors are solely responsible for the design and conduct of this study, all study analyses, the drafting and editing of the paper, and its final contents.

Limitations

Despite the strengths of using a national dataset and combining sophisticated quantitative and qualitative methodology, the study has some limitations. First, we are focusing on older patients who are Medicare beneficiaries and it is possible that the result would differ if younger patients were included; however, 80% of AMI deaths and 61% of all AMIs (49) occur in this older population and it is not clear that hospital factors associated with lower RSMRs would be different for older or younger patients with AMI. Second, CMS claims data are not available for patients who choose Medicare managed care, but this represents a minority of patients. Third, we are using claims data for our mortality model, but this model has been validated by one based on medical record data.(26) Fourth, we rely on self-reported data for the Web-based survey. We surveyed about activities that were being used at the time of the survey and therefore believe that recall bias is limited. Finally, the design precludes definitive statements about causation.

Progress to Date and Next Steps

To this point, we have completed and published the primary article describing the qualitative component of the project.(47) Further analyses of the qualitative data are in progress. For the quantitative analysis, we plan to summarize the results of the Web-based survey and report the corresponding efforts of the surveyed hospitals. These variables will be selected from the information collected in the survey regarding hospital organizational environments and their protocols and practices. For each of the strategies, we will assess the number and percentage of hospitals in each response category as well as the mean and standard deviation of RSMRs weighted by AMI admissions. We will also assess for multicollinearity and seek to combine or exclude variables appropriately. To evaluate the unadjusted associations between the independent variables and RSMRs, we will estimate Analysis of Variance models using RSMRs as the dependent variable, weighted for the number of AMI admissions included in the RSMR calculation. We will also construct a multivariable model, including independent variables that add significantly (based on likelihood ratio test P < 0.10 for nested models) to the fit of the overall model. The model will use least squares regression, weighted for the number of AMI admissions included in the RSMR calculation for each hospital.

Summary

With greater attention being directed to assessing hospital performance, there is a growing imperative to identify strategies that can promote exceptional outcomes. The SAMI study is an example of how this research is moving beyond an examination of organizational excellence in process and intermediate outcomes to an understanding of outcomes, such as AMI mortality, that are experienced by patients. The science of identifying key determinants of hospital outcomes is still nascent and much of the variation in performance remains unexplained. Methods developed by our group to understand positive deviance in health care performance are well suited to elicit strategies associated with outstanding performance. The products of this research will ideally assist hospitals throughout the country in making improvements and continuing the trend in the reductions in mortality.