Examining the Role of Amenable Mortality as an Indicator of Health System Effectiveness

Abstract

A recent paper in this journal raised questions as to the reliability, sensitivity and validity of amenable mortality as an indicator of health system effectiveness. In this commentary, we revisit the evidence that was put forward, and suggest that there are several good reasons for assessing health system effectiveness on the basis of amenable mortality. Moreover, provided multiple years of data are used to increase the precision of measurement of amenable mortality rates in very small regions, this indicator of health system effectiveness can be considered a valuable tool for research and performance measurement.

In a recent paper, Lavergne and McGrail (2013) note the growing interest internationally and in Canada in the use of amenable mortality as an indicator of health system performance, and they question the reliability, sensitivity and validity of this indicator. The purpose of this indicator is to measure, for a given population, the mortality due to causes of death before age 75 that should not exist in a well-functioning health system (i.e., the causes are amenable to health system interventions). The list of such causes of death is pre-established by experts' consensus, and there are currently several such lists (see CIHI 2012). The data necessary for the calculation of this indicator are collected through national vital statistics, each death in Canada being attributed a cause by a physician.

In this commentary, we revisit the evidence put forward by Lavergne and McGrail (2013) to demonstrate that this indicator is not reliable, sensitive and valid in a sub-provincial context. Overall, with the exception that rates of amenable mortality may be unstable in small regions, we suggest that amenable mortality is a reliable and sensitive indicator of regional health system effectiveness (“effectiveness” is a more accurate term than “health system performance” in this context), and one to which, for good reasons, health system leaders are increasingly paying attention. The next section revisits the claim that amenable mortality does not reflect the effectiveness of the health system. We then revisit the claim that it has poor statistical properties (reliability and sensitivity).

Is Amenable Mortality a Good Indicator of Health System Effectiveness?

We expect a good indicator of health system effectiveness to be sensitive to outcomes under the control of the health system: we want to measure outcomes that are affected by the way the system works and not (or rather, not too much) by the myriad factors beyond the purview of the health system. Amenable mortality is a measure of the number of deaths for which an effective cure exists (a cure that works in all cases, except for the frail). Therefore, more amenable deaths in a given region would indicate a less effective system. There is discussion regarding the meaning of “cure”: in the more stringent definition of Nolte and McKee (2004), cure means therapeutic or surgical intervention, whereas other authors would also include in the list effective prevention (making lung cancer an amenable cause of death) (CIHI 2012). Our comments and defence of the concept apply to both the stringent and the lenient definitions.

There are several reasons to support the use of amenable mortality as an indicator of health system effectiveness.

The first reason relates to the boundaries of health systems: a valid performance indicator of health system effectiveness should be sensitive to health system interventions. There are many factors outside the primary responsibility of the health system that affect death rates. Limiting the field of study to the causes of death most likely to be affected by health system interventions makes health system leaders more accountable for improvements in this measure of performance. In addition, if a large fraction of total mortality is not sensitive to the actions of the health system, using total mortality as the outcome measure will underestimate the true effectiveness of a health system relative to another one (see Box 1).

BOX 1. Measuring effectiveness.

Assume that total mortality (M) consists of two components: H, which is sensitive to the health system, and P, which is not (M = H + P), and that P represents on average 50% of M.

Now, assume that system A is twice as effective as system B: HA = 0.5*HB.

Because P is not affected by the health system, PA = PB.

As a result, MA = 0.5*HB + PB = MB - 0.5*HB = 0.75*MB.

According to total mortality, A will appear to be 33% more effective than B, whereas based on the sensitive measure, H, it is 100% more effective.

Note that if we were to measure absolute rather than relative effectiveness, we would not run into the same problem. But it would not make much sense to say that mortality (or the part of it that is sensitive to the health system) actually is effectiveness. All we want to say is that it varies with effectiveness.

A key point here is the role that social determinants of health play in explaining variations in amenable mortality, and what this implies for the indicator's validity. Lavergne and McGrail (2013) suggest that social determinants are a main driver of variations in amenable mortality, and that these undermine the validity of the indicator. We agree with the former suggestion but disagree with the latter, because social determinants of health are part of the environment explaining effectiveness. A health system's effectiveness relates both to the activities of people working in the system as well as to the environment in which they work (and influenced by the way the healthcare system is organized or funded).

For example, pneumonia is part of the CIHI's (2012) list of amenable (called “treatable” here) causes of death. Given the state of medical art today, no one younger than 75 should die of pneumonia. Smoking is a known cause of pneumonia and, as a result, a region with a higher proportion of smokers will have to devote more resources towards preventing, detecting and treating pneumonia than a region with fewer smokers; if it fails to do so, it can be considered a less effective health system. Also, if individuals living in poverty have poorer access to primary care, pneumonia may not be detected in time and, as a result, become fatal.

Regions with higher rates of poverty (or more pockets of poverty) may exhibit higher rates of (amenable) mortality due to pneumonia. Our claim is that this indicator measures effectiveness of the health system, specifically, its inability to provide access in due time to prevent death from pneumonia, even though it does not say that health professionals in regions with higher rates of poverty (or with more pockets of poverty) do not work effectively.

Evaluating the role of social determinants of health in the effectiveness of the health system is an empirical question that is worth examining. The results of such an analysis could help health policy makers and system managers identify the most effective policy levers to reduce the causes of death that are identified as most amenable to health system interventions. In other words, a regional health system that has a population with higher levels of poverty or pockets of abject poverty, for example, may need more resources or to deliver services differently in order to be effective.

Second, separating mortality into causes that are sensitive to health system interventions and those that are not allows us to use the insensitive part as a control (baseline) in statistical analyses of the effect of characteristics in the health system: if we find a correlation between such a characteristic (say, payment of physicians) and mortality across jurisdictions, we will not know whether this is pure coincidence or suggestive of a true causal effect. Now, if we find that the correlation comes entirely from a relationship between the characteristic and the part of mortality that is sensitive to the health system but not at all (correlation = 0) from a relationship between the characteristic and the part of mortality that is not sensitive to the health system, we will have a stronger inference in favour of a causal relationship between that characteristic and the effectiveness of the health system.

Finally, if we think that what Canadians expect from the health system is a guarantee of timely access and high-quality treatment when they are sick, rather than a longer-term focus on better health (Abelson et al. 2011; see Box 2), then it makes sense to use a good proxy of that guarantee when measuring the effectiveness of the health system rather than a broader measure of average population health, such as total mortality.

BOX 2. Understanding policy makers' views of the objective of the health system.

Abelson and colleagues (2011) conducted a descriptive qualitative study with key-informant interviews to understand health policy makers' views on the objectives of the health system. They conducted 17 semi-structured interviews with senior health ministry personnel from nine provinces and two territories. The stated objectives of the health system fell into two main themes:

(a) those focused on the healthcare delivery system and (b) those focused on promoting and improving the health of individuals and populations.

Healthcare delivery system objectives were mentioned more frequently than population health objectives by almost a two-to-one margin. Specifically, respondents emphasized the diagnosis and treatment of illness and disease and ensuring that healthcare is available where and when it is required as the main objectives of their jurisdiction's health system.

We agree with Lavergne and McGrail (2013) that if the component of mortality that is sensitive to health system interventions cannot be measured in a reliable way and turns out to vary exactly like total mortality across jurisdictions, it may be preferable to use total mortality as a proxy. It is important, though, to note that both conditions must hold to reject amenable mortality as a measure of effectiveness. However, if we can reliably measure amenable mortality, we should use it as an indicator of health system effectiveness because, in addition to the information that total mortality provides, it offers more detailed information on deaths considered by experts as amenable to health system interventions, and would be supported by policy makers in the health sector as an outcome measure of improvement for which they might be ready to take responsibility.

Revisiting the Claim That Amenable Mortality Is Unreliable and Insensitive

Reliability

Lavergne and McGrail (2013) use the empirical observation that in low-populated areas of British Columbia, the rate of amenable mortality varies substantially from year to year to suggest that amenable mortality is not a reliable measure. To examine this argument closely, we need to understand the relationship between the variability and reliability of an indicator.

There are three major sources of variability in a quantitative measure:

1. Sampling error: We measure a statistic on a given sample, but a different sample might yield a different value for the same statistic.

2. Random variation in the phenomenon itself: What we measure is the result of a random process.

3. Measurement error: Any time a phenomenon is observed, an error can be made in its coding or measurement.

In the case of mortality in general and amenable mortality in particular, sampling error can be ruled out: these measures are based on total population (within each jurisdiction), not on samples. Not much can be said about measurement error and, more importantly, systematic variations in such measurement error across regions in Canada, because this issue has not been studied. Moreover, Lavergne and McGrail (2013) do not raise this source of error.

Mortality is a random phenomenon: it follows a binomial distribution of parameters n (total population) and p (probability of death). As a result, the observed ratio in one year will differ from the observed ratio in another year, and we can even calculate the variance in the estimated value of the probability. Not surprisingly, the variance in the estimator increases when the total number of tries (n) decreases, and small population areas yield imprecise estimates of the probability to die of a cause amenable to health system interventions. However, imprecise is not the same as unreliable. What Lavergne and McGrail (2013) find is that amenable mortality cannot be estimated with good precision in very small regions; it is certainly less precisely estimated than total mortality, because the probability that we try to estimate is smaller. As a result, we cannot rely on empirical estimates of amenable mortality (nor, for that matter, of total mortality) based on small numbers of tries. Analysts who want to evaluate the effectiveness of a health system using amenable mortality must make sure they use enough years of data to get a more precise estimate, or that they account for random variation in their conclusions on effectiveness. But, again, the same would be true with any stochastic outcome measure.

Insensitivity

Lavergne and McGrail (2013) use empirical evidence that amenable mortality is highly correlated with premature all-cause mortality across 16 health regions in British Columbia, and we confirm the strong correlation between these two measures (amenable and all-cause premature mortality rates) across 90 health regions in Canada's provinces (with a correlation coefficient of 0.96).¹ However, we disagree on the inference drawn from the empirical evidence.

First, we argue that such a strong correlation provides support for the reliability of amenable mortality: if there were systematic error in the measurement of amenable mortality due, for example, to difficulties in coding the primary cause of death on death certificates, then we would see weaker correlations across health regions between the two variables.

Second, and more importantly, because the two measures are not perfectly correlated, there is additional information that can be gained by examining variations in amenable mortality that all-causes mortality would not yield.² We can learn something from the fact that Vancouver does less well on amenable mortality than its all-causes mortality ratio suggests. Lavergne and McGrail (2013) read such a discrepancy as further proof that amenable mortality is not a valid measure of effectiveness: because it is well known that Vancouver has more doctors and hospitals per capita than other regions, its health system should be more effective and, as a result, if amenable mortality indicates it is less effective, this suggests that the indicator is not reliable or valid. We respectfully disagree with this conclusion: in our opinion, the high level of amenable mortality in Vancouver shows that despite its high density of physicians and hospitals, the effectiveness of the health system is not only a matter of resource availability and usability, but also the product of the needs and demands of the population. The population of Vancouver is healthier than the rest of British Columbia but at the same time, those in Vancouver who suffer from diseases that are amenable to health system interventions are perhaps sicker or have more complex needs and require more resources per case (as, for instance, in the already mentioned case of pneumonia, some patients in Vancouver may have issues in accessing primary care). Through this example, we see that amenable mortality tells us something that all-cause mortality cannot tell, but also something that the supply of health services cannot tell, either. In particular, health system managers in Vancouver would benefit from knowing the particular causes of death that drive their high level of amenable mortality.

Conclusion

Lavergne and McGrail (2013) seem to reject amenable mortality on the grounds that (a) it focuses on secondary and tertiary prevention to the detriment of primary prevention, and (b) it implies that too great an investment in primary prevention may be less effective. We fully agree that measuring preventable mortality is of interest to policy makers because it can help identify areas where attention should be paid. However, the key question is whether we should use amenable or preventable mortality as a measure of health system effectiveness: this leads us back to our question about what Canadians expect from the health system. If we think that the outcome of interest is a healthier population on average, then effectiveness should be measured as the reduction of preventable mortality; however, if we think that the outcome of interest is timely access to high-quality care when needed, then amenable mortality is a better proxy for effectiveness. Based on the results of recent research commissioned by CIHI (Abelson et al. 2011), amenable mortality seems to be a more appropriate measure of health system effectiveness, and one for which policy makers might be ready to take responsibility to support improvements in health system outcomes.