Some notes on critical appraisal of prevalence studies

Abstract

Decisions in healthcare should be based on information obtained according to the principles of Evidence-Based Medicine (EBM). An increasing number of systematic reviews are published which summarize the results of prevalence studies. Interpretation of the results of these reviews should be accompanied by an appraisal of the methodological quality of the included data and studies. The critical appraisal tool for prevalence studies developed and tested by Munn et al. comprises 10 items and aims at targeting all kinds of prevalence studies. This comment discusses the pros and cons of different designs of quality appraisal tools and highlights their importance for systematic reviews of prevalence studies. Beyond piloting, which has been performed in the study by Munn et al., it is suggested here that the validity of the tool should be tested, including reproducibility and inter-rater reliability. It is concluded that studies as the one by Munn et al. will help to establish a critical understanding of methodological quality and will support the use of systematic reviews of non-intervention studies for health policy making.

During recent years, it has become more and more accepted that decisions in healthcare have to be based on information obtained according to the principles of Evidence-Based Medicine (EBM) (1,2). Traditionally, EBM methodology has focused on systematic reviews of intervention studies. However, an increasing number of systematic reviews are published which summarize the results of prevalence studies, and specific techniques have been developed for meta-analysis (synthesis) of this kind of data (3).

Systematically searching and reviewing evidence on prevalence of disease is important for a variety of reasons:

• Prevalence studies are used to inform researchers, guideline developers and policy-makers about burden of disease, thereby supporting the process of identification of priorities in healthcare, prevention and policy.

• Prevalence studies are needed for the development of health economics models. It has become more and more accepted that modelling studies should be conducted to assess the economic impact of an intervention before its implementation in a patient group or population. Input parameters needed for modelling studies such as the prevalence of a disease or complication can have a great impact on the results.

• Prevalence studies inform the assessment of interventions, since they provide data on the baseline risk for a given disease in a patient group or population which influences effect measures (4).

However, evidence does not speak for itself (5). It needs to be interpreted critically, with reviewers being aware of the strength and limitations of the data. Interpretation should be accompanied by an appraisal of the methodological quality of the data and studies. Therefore, a variety of tools and checklists have been developed to support the process of critical appraisal of single studies [for an overview, see (6–9)]. In this context, it is important to clarify the definition of ‘methodological quality’. The Agency for Healthcare Research and Quality (AHQR) defines methodological quality as the extent to which a study’s design, conduct, and analysis has minimized selection, measurement, and confounding biases (8). Applying this definition, the term “methodological quality” is used interchangeably with “risk of bias” or “internal validity”. However, many critical appraisal tools also incorporate items related to external validity, which might be important if results from a study should be transferred to another country or setting.

The new tool for the critical appraisal of prevalence studies developed and tested by Munn et al. (10) comprises 10 items. In addition, the authors provide an appendix with detailed descriptions and explanations which will be helpful for those users who are not familiar with critical appraisal tools. The tool includes items related to the internal validity of the study (e.g. were study participants recruited in an appropriate way?), but also some on external validity (e.g. was the sample representative of the target population?). If one compares the tool to those which are already available for assessment of prevalence studies, some similarities but also differences become apparent. While the number of items/criteria used is very similar to other tools [e.g. Hoy et al. (11): 10 items; Loney et al. (12): 8 items], the tool by Munn et al. strictly focusses on validity and does not assess reporting quality, which is incorporated in some other tools such as the one developed by the National Collaborating Centre for Environmental Health (NCCEH) (13). In general, it is important to distinguish between methodological quality and the transparency and quality of reporting (9) since it has been shown that studies with similar reporting quality may be different regarding methodological quality (14). Furthermore, the tool by Munn et al. differs from others regarding the study types to be covered. A recent methodological study concluded that quality appraisal tools are often designed to address a certain study type, rather than a research question (8). For example, the consequences of an intervention can be either studied by Randomized Controlled Trials (RCTs) or observational studies. In RCTs, blinding of participants and observers is an important issue which is addressed by many critical appraisal tools, such as the Cochrane Risk of Bias Tool (15). Without adequate blinding, the validity of the trial can be seriously damaged. However, the same intervention effects can also be addressed by other study types, such as non-randomized controlled trials, or even cohort studies or case-control studies, although these study designs are not ideal for these questions. For different study designs, other issues which are specific for those study designs become important, such as the process of selection of controls. They are addressed by tools specifically designed for case-control studies, such as the Newcastle-Ottawa Scale (NOS) (16). However, some quality appraisal tools exist which aim to cover a large variety of study designs, such as the one developed by Cho et al. (17) which was designed to cover intervention as well as non-intervention studies. The tool reported by Munn et al. aims at targeting all kinds of studies reporting prevalence data. It remains to be investigated whether tools designed to cover a specific study design provide more transparent results and are easier to apply than those critical appraisal tools that aim at covering more than one study design.

Two subsequent steps are important when a new quality appraisal tool has been developed. First, it is necessary to pilot the tool, in order to test its applicability and user friendliness. This step has been performed by Munn et al. and the results look promising, as described in their paper. The second step, however, is of equal importance. The validity of the tool should be tested, including reproducibility and inter-rater reliability. In fact, only a minority of the so far published quality appraisal tools has been validated (8), but validation is important for further improvement of the tools (16,18). Munn et al. should consider performing such a validation study which will surely support the acceptance of their tool by the scientific community.

Last but not least, more studies like the one reported by Munn et al. are needed on the usage and applicability of critical appraisal tools. Such research will help to establish a critical understanding of methodological quality and will support the use of systematic reviews of non-intervention studies for health policy-making.

Ethical issues

Not applicable.

Competing interests

Author declares that he has no competing interests.

Author’s contribution

TH is the single author of the manuscript.

Citation: Harder T. Some notes on critical appraisal of prevalence studies; Comment on: "The development of a critical appraisal tool for use in systematic reviews addressing questions of prevalence". Int J Health Policy Manag 2014; 3: 289–290. doi: 10.15171/ijhpm.2014.99