Evaluating the effectiveness of public health interventions: the role and activities of the Cochrane Collaboration

Abstract

Public health decision makers, funders, practitioners, and the public are increasingly interested in the evidence that underpins public health decision making. Decisions in public health cover a vast range of activities. With the ever increasing global volume of primary research, knowledge and changes in thinking and approaches, quality systematic reviews of all the available research that is relevant to a particular practice or policy decision are an efficient way to synthesise and utilise research efforts. The Cochrane Collaboration includes an organised entity that aims to increase the quality and quantity of public health systematic reviews, through a range of activities. This paper aims to provide a glossary of the terms and activities related to public health and the Cochrane Collaboration.

Public health decision makers, funders, practitioners, and the public are increasingly interested in evidence to underpin public health decision making. With the ever increasing global volume of primary research, knowledge, and changes in thinking and approaches, quality systematic reviews of all the available research relevant to a particular practice or policy decision are an efficient way to synthesise and utilise research.

Research relevant to public health is not isolated to evidence of the effectiveness of interventions, however interventions need to be researched and evaluated for effectiveness, economic, ethical, and accountability reasons. Cochrane Collaboration systematic reviews are one source of analysed evidence available to all public health decision makers (managers and public policy makers, clinicians, health promotion and public health practitioners, and researchers). Cochrane systematic reviews identify and synthesise quality research to produce reliable, transparent answers to questions of the effectiveness of interventions. They are published in an electronic library format and updated regularly, should new evidence become available.

This paper briefly describes the issues for Cochrane Collaboration systematic reviews for public health and activities to improve the quality and quantity of reviews for public health decision makers. A previous glossary, by Rychetnik et al,¹ has already defined many of the concepts associated with evidence based public health. Therefore, in this paper we describe the processes and challenges of the Cochrane Collaboration and public health and health promotion.

Navigating the Cochrane collaboration

The work of the Cochrane Collaboration is carried out by over 70 entities. Each entity is responsible for identifying their own source of funding to support their programme of work. The programme work across the collaboration is varied, from carrying out or peer reviewing a review, to enhancing methods, increasing their relevance and usability, identifying priority reviews, collecting studies, and handsearching journals. The collaboration's direction is determined by a Steering Group, with several working sub‐groups, aided by an Ombudsman, as well as a Funding arbiter and a Publication Arbiter for issues surrounding minimisation of bias and resolving matters of conflict.

Collaborative Review Groups (CRG) are responsible for the production and maintenance of Cochrane reviews, from title registration to publication of the completed review on the Cochrane Library, and were formed originally around disease groupings, moving to policy related groups over time, for example, Communicating with Consumers, Effective Practice and Organisation of Care. Review authors align themselves with a topic appropriate CRG that will edit, peer review, and publish their review. Reviewers are also supported by their relevant Cochrane Centre (country based), which provides training and administrative support, Methods Group, which provide advice regarding review methodology, and Fields and Networks who provide support and information regarding Cochrane processes for reviewers, and engage with users and stakeholders to promote the Cochrane Library resource and address issues of quality and relevance, in their particular area of interest.

Cochrane reviews process

The review title must first be registered with one of the CRGs who will ensure that the topic is new. A Cochrane protocol is prepared by the review author, peer reviewed by the CRG, and published on the Cochrane Library. The protocol outlines the research plan in advance, thus reducing bias that could be introduced once the review is in progress. Production of the Cochrane review then proceeds as outlined in the protocol, is peer reviewed, edited, and published on the Cochrane Library as a completed review.

The cochrane health promotion and public health field

In 1996, four years after the start of the Cochrane Collaboration, a health promotion oriented Field started. The Health Promotion Field was later extended in scope to include public health (details of the history are on the HP&PH Field's web site: http://www.vichealth.vic.gov.au/cochrane). For the purposes of determining a scope of responsibility within the Cochrane Collaboration, the Cochrane Health Promotion and Public Health Field developed the following working definition, based on existing and well published definitions: “Health promotion and public health strategies are based on the understanding that health is a concept engaging social, mental, spiritual and physical well‐being. Public health encompasses the assessment of the health of populations, formulating policies to prevent or respond to health problems, promoting healthy environments, and generally promoting health through the organised efforts of society. Public health promotes societal action to invest in living conditions that create, maintain and protect health.” This covers an extremely wide range of interventions aimed at improving health, with various levels and types of intervention included. In general, the Cochrane HP&PH Field supports and encourages systematic reviews of interventions that have, as its focus, target groups or populations defined by sociodemographic characteristics, or higher risk for a poor health outcome.

The Cochrane HP&PH Field seeks to represent the needs and concerns of health promotion and public health policy makers, practitioners, researchers, consumers, and reviewers within the processes of the Cochrane Collaboration. Effectively, this entails all aspects of the promotion, production, and use of systematic reviews of effectiveness of health promotion and public health interventions.

Our strategic work programme aims to address the methodological and content related aspects of public health systematic reviews, as well as strategies required to ensure the reviews are appropriate to public health stakeholders. Four priority projects are currently underway: global priorities for public health review topics (in partnership with global public health organisations such as the World Health Organisation, the Global Health Council, and others)²; developing guidelines for conducting public health systematic reviews (in partnership with experienced public health systematic reviewers and researchers)³; developing a comprehensive framework for locating public health research,⁴ and developing a comprehensive trial and study register. We publish regular articles in international peer reviewed journals and develop Memoranda of Understanding with appropriate organisations, for example, WHO.

Cochrane reviews in public health

Advantages of Cochrane reviews in public health

Unlike most journals that publish peer reviewed research, the editorial facilities of the Cochrane Review Group's provide peer reviewed feedback at the protocol stage of the review research process. Guidelines for reviewers, a standard format, and peer review help to ensure reviews are carried out and presented in a similar way, to improve their readability and interpretability, ensure quality, and reduce bias. To ensure relevance to end users, a process exists for consumer input in both the formulation, writing, and critiquing of the review. Cochrane reviewers make a commitment to regularly update their reviews to ensure the latest research is incorporated. Cochrane reviews are accessible electronically through the Cochrane Library or in CD format, and many countries now hold national licences, making them freely available to the user.

What this paper adds

This paper illustrates the complexity and appropriateness of using the Cochrane Collaboration organisational and communication capacity to conduct public health systematic reviews.

Reporting of the methods of health promotion effectiveness reviews are generally poor, making it difficult to assess potential bias, and hence the validity and comprehensiveness of the review's findings.⁵ In general, Cochrane reviews are an exception. Empirical evidence has shown that Cochrane reviews are, on average, more systematic and less biased than systematic reviews published in paper journals.⁵,⁶,⁷,⁸ Irrespective, Cochrane reviews should be read critically as errors and biases may still occur,⁹ however, the Cochrane Collaboration aims to continually address these issues through a range of organisation wide quality management and quality control processes. Processes and methods to continually improve health promotion and public health reviews are being addressed through guidelines and engagement with key stakeholders and users.

Misconceptions and challenges with Cochrane reviews in public health

There are diverse views on what constitutes evidence of health promotion effectiveness, including the appropriateness of randomised controlled trials for evaluating health promotion and public health interventions. Cochrane reviews primarily focus on randomised controlled trials.¹⁰ However, the Cochrane Reviewers' Handbook states that “while attention to the risk of bias should guide decisions about what types of study designs to include in a review, individual reviewers and Collaborative Review Groups must decide what types of studies are best suited to specific questions”.¹⁰ The Non‐randomised Studies Methods Group is currently preparing guidance on the use of non‐randomised studies in Cochrane Reviews.¹¹ The issue that has concerned the statisticians is “how far is it possible to achieve a higher level of relevance by including evidence other than that derived from randomised controlled trials without violating the central principle of minimising bias”?¹²

There is also a widely held myth that Cochrane systematic reviews can only be used for very narrow biomedical interventions focused on individuals.¹³ For example, Cochrane reviews have been described as having a very narrow scope,⁵,¹⁴ limited to relatively simple interventions,¹⁴ and only being concerned with secondary prevention issues.⁵

Increasing the scope of reviews in public health

Cochrane reviews address questions about the effects of healthcare interventions,¹⁰ however this is interpreted broadly: educational and psychosocial primary prevention interventions are included in this scope; and population as well as individual outcomes can be included. Review authors are now challenging and addressing the concerns and misconceptions raised above. Examples of review titles include: multiple risk factor interventions for primary prevention of coronary heart disease; population based interventions for reducing sexually transmitted infections, including HIV infection; area wide traffic calming for preventing traffic related injuries and strategies for integrating primary health services in middle and low income countries.

The increasing policy driven need to focus on questions associated with educational, social, and criminological interventions has stimulated the development of the Campbell Collaboration (http://www.campbellcollaboration.org) that, through similar mechanisms, seeks to conduct systematic reviews in these areas. The Campbell Collaboration is still in its early years however there are clear opportunities for cross collaboration, for example, equity methods, social care, and social policy reviews (with health and social outcomes). The Steering Groups of both organisations now share common members to strengthen these collaborative possibilities.

Resource requirements

Cochrane reviews entail rigorous resource intensive processes to strengthen quality and minimise bias. This includes searching published and unpublished literature, including studies from all languages, the engagement of at least two researchers to screen and extract data from each primary study, processes to compare and contrast results from each researcher and subsequently resolve disagreements, checklists to examine primary study quality, identification of all publications pertaining to each intervention (and in some cases the data varies substantially). New guidelines for public health reviews have the potential to substantially improve the content however it remains to be seen whether this will increase or decrease resources required.³

Complexities and challenges of reviews in public health

Issues and difficulties that arise and need to be taken into account when synthesising results of multiple studies are: (1) a focus on populations and communities rather than individuals; (2) difficulties characterising and simplifying complex multi‐component interventions rather than single interventions; (3) an analysis of process as well as outcome measures; (4) the effect of involvement of community members or potential participants in programme design and evaluation; (5) the effect of using health promotion theories and beliefs; (6) an analysis of the use of different types of both qualitative and quantitative research; (7) the need for multiple primary papers that may cover the complexity and long term nature of public health intervention outcomes,¹⁵ and (8) integrity of the intervention highlighting what factors may have influenced the effectiveness (or ineffectiveness) of the intervention, such as participation (including appropriateness), exposure of programme or intervention, resources, quality of delivery (including training and enthusiasm), and safeguards against intervention contamination.²⁵

For these reasons, conducting systematic reviews of all the available evidence can be a complex task, necessitating reviewers to have (or have access to) sound content and methodological knowledge and expertise.

Evidence of effectiveness

The methods used to provide evidence of effectiveness must be sufficiently comprehensive to encompass the complexity of public health interventions.¹⁶ The study designs favoured by the Review Groups within the Cochrane Collaboration, in order to answer questions of effectiveness, have an intervention and control (or comparison). Therefore the methods eligible for inclusion in Cochrane reviews are generally randomised controlled trials (RCTs), quasi‐randomised controlled trials, non‐randomised controlled trials, and interrupted time series designs. Uncontrolled studies are generally not included in reviews as it is difficult to distinguish the effects of the intervention from the Hawthorne effect or from what would have occurred naturally over time. However, in many areas of public health RCTs may be uncommon, as RCTs tend to be suited to more simple and straightforward interventions. There are some particular exclusions; methods used in the systematic review of the effectiveness of vaccines on their harms and benefits use cohort, case‐control, and case‐only designs. This has also been trialled with the cross country evaluation of mass media strategies.

Randomised controlled trials and quasi‐randomised controlled trials refers to trials where participants or populations are randomly allocated (for example, computer generated randomisation, random number table) to an intervention or control/comparison group and are followed up over time to assess differences in outcome rates.¹⁷ A quasi‐randomised trial uses a method of allocation that differs from genuine randomisation for methodological (for example, allocation by date of birth, alternate allocation)¹⁷ or pragmatic and policy reasons (for example, allocation by housing sector).

Non‐randomised controlled studies (controlled before and after studies) refers to a study design where participants or populations are allocated by the investigator, in a non‐randomised fashion, to an intervention or control group. The outcome of interest is measured both at baseline and after the intervention period, comparing either final values if the groups are comparable at baseline, or if not, changes in outcomes.¹⁷ The lack of randomisation in these types of studies may result in groups being different at baseline, as randomisation is the only way to control for confounders that are not known or not measured.¹⁰

Interrupted time series designs are “multiple observations over time that are ‘interrupted' usually by an intervention or treatment”.¹⁸ These designs may or may not include a control group.

Process evaluations (often published separately from outcome evaluations) may also be included in the review, alongside quantitative studies, to assess the adequacy of the delivery of the intervention, and the context in which the intervention was evaluated. Process data have conventionally been drawn from observational quantitative research but increasingly use qualitative and quantitative research methodologies, as appropriate.

Process evaluation is a study of the process of the delivery of the intervention. They can be used to help disentangle the factors that are responsible for successful outcomes, implementation of the intervention, and intervention integrity. Process data assists with characterising the failure to achieve success¹⁹ (intervention integrity—see below).

Searching for public health literature

Retrieval of information in clinical medicine is facilitated by clinical medical literature being comparatively well organised, comparatively easily accessible through large sophisticated bibliographical databases, domination of the peer reviewed journal format, and comparatively well controlled and stable technical terminology.²⁰ Retrieval in public health is much more complicated because of a more diverse literature (reflecting the multi‐disciplinary nature), a wider range of bibliographical tools of varying coverage and quality, and terminological difficulties.²¹ Identifying public health studies is also problematic because of database indexing, as many studies may not be well indexed, or indexed differently between databases.²⁰ Moreover, a great deal of public health research is widely dispersed, and may not always be available in the public domain.²² In a review of the effectiveness of crime prevention interventions Casteel and Peek‐Asa²³ obtained more than half of their articles from reviewing references and talking to contacts. Projects are currently underway to provide recommendations for searching public health and health promotion literature. The EPPI‐Centre, in London, provides support and resources to assist review researchers conduct sensitive searches (http://eppi.ioe.ac.uk) The HP&PH Field has nearly completed a project that will provide recommendations for search terms and hand searching strategies, for published and “grey” literature.^23a

Critical appraisal of public health interventions

Critical appraisal of intervention research entails the assessment of validity of the evidence, the completeness of implementation (intervention integrity) and applicability of the evidence. There are many quality tools or checklists available to help the reviewer assess the extent to which the methodology of the study sought to minimise bias. At present there are no commonly agreed criteria for assessing study quality, as no strong evidence of a relation between trial outcomes and specific quality criteria exists. Thus different systematic reviews use different sets of quality criteria. Appraisal of public health studies is particularly challenging because of (1) difficultly in blinding participants to certain interventions (particularly educational initiatives), (2) the potential for the control/comparison group to become “contaminated” (for example, within schools where participants in the intervention and control groups are highly likely to come into contact with each other) and (3) the potential threats to the validity and reliability of data collection methods, particularly where outcomes are subjective (for example, reported behaviour). In addition, incomplete reporting of vital study information hinders a complete assessment of study quality.²⁴

For these reasons, experienced public health research reviewers advocate that, as a minimum, public health reviews should assess each included study to determine: complete reporting of number of participants in control and intervention groups; complete reporting of pre‐test and post‐test data for all participants in both groups; and the provision of complete data for all outcomes. This minimum amount of information is needed before studies can be further assessed for random allocation and blinding of participants.²⁴

Synthesis of results

Public health interventions are often difficult to synthesise because of the complexity of the characteristics of the intervention, the study population(s), the outcomes measured, or other methodological issues (including study design) relating to the conduct of the primary studies.²⁶ Furthermore, complexity is introduced because the effectiveness of the interventions may be modified by the context in which it operates.¹⁶,²⁶ Because of the potential for variability of the characteristics between studies, reviewers may choose to use a narrative synthesis of results, as calculation of a statistical overall estimate would not be meaningful as one would be comparing “apples with oranges”. Where the interventions are homogenous, and outcomes comparable, meta‐analysis remains a useful tool to synthesise outcomes of multiple studies, however meta‐analysis may be generally inappropriate because of the degree of heterogeneity between studies.

Narrative synthesis (also referred to as descriptive data synthesis) is a qualitative assessment of the variation of study characteristics, quality, and results²⁷—that is, dissimilar studies, the findings from the range of studies included in the review are summarised and explained in words.

Policy implications

Policy makers and practitioners can now be confident that processes are in train to ensure that public health reviews conducted and published by the Cochrane Collaboration are moving towards a format and content appropriate to their needs. Developments in priority setting, guidelines for systematic review research, and engaging stakeholders in the process have been core developments for this outcome. Over the next few years a number of priority reviews will be conducted and published that will be essential tools in global health decision making.

Applicability

The process of determining how the results of a review relate to another specific situation, context, or intervention is called applicability, transferability, or generalisability. These terms are essentially synonymous with external validity. This information provided in reviews is particularly relevant to users and their decisions to enable them to assess the applicability of the results to their individual settings.

Systematic reviews of public health interventions encompass a number of issues that may complicate the process of determining applicability. Less well defined inclusion criteria in some public health studies, determination of the effect of individual components of a complex intervention, and the measurement (and reporting) of implementation and compliance of community interventions can all contribute to the difficulty in interpreting and applying the findings from systematic reviews. Systematic reviews that include a number of studies (with consistent results) that have been conducted in a range of settings, would suggest wide applicability.²⁸

Context refers to the social, organisational, and political setting in which the intervention is implemented. Examples of contextual factors that may affect intervention effectiveness include literacy, income, cultural values, and access to media and health services.²⁸

Ethics and inequalities

The benefits of many public health interventions may not be evenly distributed, and in some cases, interventions may exacerbate health inequalities. For these reasons, the Health Promotion and Public Health Field, through its current guideline initiative, is bringing together background literature in the area of public health ethics to raise the debate regarding identifying and discussing whom the intervention is benefiting, and for whom the intervention may cause harm.

Until recently, Cochrane reviews have largely paid poor attention to issues related to intervention equity and inequalities.²⁹,³⁰ Collaboration members involved in public health, child health, and some Review Groups with particular expertise in inequalities research, have continued to advocate, through the conduct of specific reviews on the differential impact of interventions, that specific methodological developments are warranted. This will include the development of assistance to extract and synthesise primary data that describe the ability of interventions to reduce population inequalities, or adversely increases inequalities.