Ecological studies: use with caution

BACKGROUND

Ongoing debates about the quality of NHS organisations have made ecological studies fashionable. One such study in the UK considered the association between the average clinical quality of primary care provided by primary care trusts and the trusts’ rate of admission for coronary heart disease.¹ An alternative to this ecological approach, which used data aggregated to the level of a primary care trust, would have been to have used data for individual patients, and to have asked if there was an association between the clinical quality of care an individual receives and their own chance of being hospitalised for coronary heart disease. By using aggregated data in ecological studies the relationship for individual patients is not directly explored, although individual relationships may often (correctly or incorrectly) be inferred from population-based analyses. Ecological studies can either be descriptive, for example, exploring variation between populations, or consider associations such as the example above. In health services research, where healthcare organisations rather than individual patients are often the focus of inquiry, ecological studies are often an appealing tool. For example, a recent study looking at the features of general practices associated with lower coronary heart disease mortality was more concerned with the practice at an organisational level than with individuals.²

STRENGTHS OF THIS APPROACH: OPEN DATA AND ORGANISATION ANALYSES

The availability of data describing NHS organisations has never been greater. A large volume of UK healthcare process and outcome data is becoming publicly available from the Health and Social Care Information Centre (http://www.hscic.gov.uk/) and via the government open data website (http://www.data.gov.uk/) with GP practice, hospital, and clinical commissioning group (CCG) indicators available. Indicators cover measures including population, service, clinical outcome, prescribing and patient experience. The UK Data Archive (http://data-archive.ac.uk/) is another source of publicly available data, including individual level data from health surveys. With the UK white paper from 2012 presenting the government strategy to make more data public³ the amount of data available is only going to increase.

One strong advantage of using publicly available data in research is that there are no problems with data confidentiality. When looking at associations between risk factors and outcomes, linking information about individual patients often requires extensive ethical and governance approval. Linking data at the organisational level, however, does not, as usually the data have been published and are in the public domain already. Ecological studies also allow us to look nationwide providing evidence that is potentially more generalisable than from studies considering individuals, but in only a small geographical area. A further strength of ecological studies is that where data are available the exploration of potential trends over time can be considered with relative ease.

The strength of an ecological study for looking at associations at the institutional level (hospital, CCG or GP practice) is balanced by the fact that we cannot draw conclusions about individual patients from population data. We can tell whether GP practices where patients report a better experience of care are those that achieve higher QOF targets,⁴ but this tells us nothing about the association between patient experience and the quality of clinical care at the patient level. Ecological studies are at best hypothesis generating when considering individual level associations and care is needed to avoid the risk of ecological fallacy: assuming the associations that exist at the population level persist at the individual level. Ecological analyses which consider within-institution trends-over-time are less vulnerable to these problems, but not immune.

CAUTIONS: CONFOUNDING, BIAS, AND ECOLOGICAL FALLACY

Considerations applicable to any type of epidemiological research also apply to ecological studies, for example in relation to potential confounding (where two domains of care appear associated, but this is in fact simply because they are both associated with a third, confounding, variable). Where there is confounding by individual level variables (such as clinical diagnosis or disease severity case mix or sociodemographic variation), then if individual level data are available for one of the measures of interest, accounting for potential confounding at the ecological level is possible.⁵ Information may also be available at the organisational level about possible confounders, but individual level data are needed if individual level confounders are the concern.

The importance of these last two points (potential for ecological fallacy and unmeasured confounding), when interpreting correlations observed at the organisational level, cannot be overstated. A simple, unadjusted, correlation of two measures at the population level has the potential for eye-catching headlines, such as the study of the association between chocolate consumption and winning a Nobel prize.⁶ However, the potential for ecological studies to lead into suboptimal policy-making is high; confounding and ecological fallacy mean that an unthinking analysis of associations at the organisational rather than the individual level may have far reaching consequences. Recently it has been claimed that NHS hospitals that operate in a more competitive geographical environment have a lower mortality rate for patients with myocardial infarction.⁷ Whether this association was causal or not has been a subject of a lively debate.⁸

Data completeness is also important. Complete and accurate data is incentivised in the NHS, but there remains variation in quality and validity across organisations. For example, exception reporting varies across GP practices in the UK⁹ and there is considerable variation in data quality in hospital-acquired infection surveillance.¹⁰ Measurement bias (where errors in data measurement are associated with healthcare organisation performance) can also be a concern even using standardised publicly reported data. Further, where data is sparse, confidentiality requirements in the public reporting of data means that information is suppressed in public sources where it may be individually identifiable; for example, data may be disproportionately more likely to be missing for single-handed GP practices.

FURTHER CONSIDERATIONS: POWER AND RELIABILITY

Other methodological questions should also be considered. The statistical reliability of the measures in question at the organisational level are important to consider.¹¹ Additionally, if several comparisons are being made then statistical tests should be adjusted for multiple testing. The temptation to start correlating everything with everything else, just because the data are freely available and accessible, should be avoided and analyses should be hypothesis-led wherever possible.

Analyses also need to be adequately powered. For example, given there are only around 160 hospitals in England, a study using all of these would have 80% power to detect a correlation of 0.22. While this would not be described as a strong correlation it is larger than values often found in ecological studies. The fact that only relatively strong associations will ever be detected by ecological studies of this sample size potentially encourages the publication of false-positive results as any statistically significant finding accompanies a large effect size. Similar cautions apply to ecological studies in general practice settings when only a small geographical area is considered (for example, within a CCG). Additionally, if the measurement of organisation performance does not have high reliability then power will be further decreased.

BEST PRACTICE AND CONCLUSIONS

The need for good practice in working with and reporting health services research carried out using routine health data are clearly wider than the epidemiological concerns about the ecological study design alone. The RECORD (the REporting of studies Conducted using Observational Routinely-collected Data) statement, an extension of STROBE, (STrengthening the Reporting of OBservational studies in Epidemiology) is in development, defining reporting guidelines for observational studies using health data routinely collected for non-research purposes.

Ecological studies in health services research are a powerful tool and with the wealth of organisational level data now available, there are increasing numbers of research questions where they are the study design of choice. However, the potential for over-interpretation of results and generation of spurious findings is ever present. Good practice in the use of routine health data for research and the use of standard epidemiological precautions are necessary when carrying out and interpreting these studies.

Provenance

Freely submitted; not externally peer reviewed.

Competing interests

The authors have declared no competing interests.