Table 1.

Principles, strengths, weaknesses and opportunities for study designs for improvement interventions

Class of studies		Principles	Strengths/weaknesses	Opportunities for methodological improvement	Example
Quality improvement projects		Project is set up primarily as an improvement effort, to learn what works in a local context. It is typically motivated by a well-defined problem and oriented towards a focused aim. PDSA cycles are often applied, allowing for testing incremental, cyclically implemented changes, which are monitored through statistical process control	Strengths: flexibility in testing changes and adapting interventions; incorporation of knowledge generated by local improvement experiences; ability to interactively move from testing the QII locally to applying it more broadly. Weaknesses: generalisability of findings is not straightforward; lack of structured explanation of mechanisms of change; frequent low quality of reports	Quality improvement projects should incorporate theoretical base and qualitative methods more systematically to allow for predicting and explaining the mechanisms of change involved; more scientific vigour is needed in the application and reporting of PDSA cycles and other methods/techniques applied	An improvement initiative based on social marketing interventions developed to increase access to a psychological therapy service (especially from areas of high deprivation) involved weekly collection of geo-coded referral data and small-scale tests of change57 58
Effectiveness studies	RCTs	RCTs may be especially suitable whenever interventions are being considered for widespread use based on their face validity and early or preliminary evidence. Differences in outcomes from delivering two or more interventions to similar groups of people or other entities are attributable to differences between the interventions. Control of confounding factors is an explicit aim	Strengths: direct inferences on causality. Weaknesses: neglect the weak boundaries separating context and intervention and the multiple interactions that take place between them; randomisation and blinding may be difficult or even not applicable; risk of contamination between groups	Improvements in the design, conducting, and reporting of RCTs are necessary to limit the high risk of bias observed currently. The awareness of the value of robust design, the need to avoid preconceived judgments about the intervention, and investments in research methods training should be pursued	A study aimed to determine the causal effects of an intervention shown effective in former pre/post studies in reducing central line-associated bloodstream infections in intensive care units.72
	Quasiexperimental designs	The intervention is implemented and followed-up over time, ideally with a control. Compared with a RCT, the investigator keeps more control over the intervention, but has less control over confounding factors	Strengths: often more practical to conduct than an RCT. Weaknesses: causality is not inferred directly, and confounding factors’ effects may not be obvious	Whether they have controls or not, quasiexperimental studies will be more powerful if they involve multiple measurements before and after the intervention is applied	A before-after study with concurrent controls sought to evaluate an intervention to reduce inpatient length of stay and considered the effect of the reduction on patient safety80
	Observational (longitudinal) studies	The implementation of the intervention is observed over time	Strengths: cases in practice may be the focus of the study; may be especially useful in the evaluation of sustainability of interventions. Weaknesses: inferences about causality may be challenging	Can be useful when other studies are not possible. They must be longitudinal and, ideally, prospective. The absence of an explicit control in the study design may be compensated by statistical techniques	A study aimed to examine the sustainability of an in-hospital quality improvement intervention in AMI, including the identification of predictors of physician adherence to AMI-recommended medication87
	Systematic reviews	Combining findings/samples from RCTs and quasiexperimental studies on the effectiveness of an intervention allows for more robust and generalisable QII effectiveness results	Strengths: ability to generate more powerful evidence. Weaknesses: uncritical incorporation and interpretation of studies may lead to inadequate conclusions; low use of meta-analyses	The development of systematic reviews on the effectiveness of QIIs has grown. It needs more critical appraisal of the studies included, more meta-analyses, and to deal with complex interventions in diverse contexts	Systematic review with meta-analysis aimed at assessing the effects of QIIs on the management of diabetes88
Process evaluations		Understanding what an intervention is in practice important, especially when the aim is to attribute effects to it	Strengths: process evaluations make possible an understanding of improvement interventions in practice and the fidelity with which they are implemented	Process evaluations should be embedded in effectiveness studies to capture failures in the QII implementation, and to better understand how QIIs’ components act. They need also to be more oriented towards validating theory-informed strategies	Process evaluation of a cluster randomised controlled trial aimed to examine which components of two hand hygiene improvement strategies were associated with increased nurses’ hand hygiene compliance70
Qualitative studies		It is not enough to know that an expected change happened or did not. It is important to understand why and how	Strengths: ability to capture, considering different points of view, the extent that interventions are implemented, and to explain mechanisms of change involved, based on theories	Qualitative studies should be included in quality improvement projects and QIIs’ quantitative evaluative studies for better understanding of outcomes and explanation of mechanisms of change involved.	Study that developed an ex post theory of the Michigan Intensive Care Unit project to explain how it achieved its effects101
Economic evaluations		It is important to know that an intervention is effective and also that the investment required is justifiable	Strengths: adds information about how justifiable the QII is in face of the investment required	In the literature, studies dedicated to economic evaluations of healthcare QIIs are still lacking, and there is recognition that there should be more of them in the field	Cost-effectiveness analysis of a multifaceted intervention to improve the quality of care of children in district hospitals in Kenya104

AMI, acute myocardial infarction; PDSA, Plan-Do-Study-Act; QII, quality improvement intervention; RCTs, randomised controlled trials.