Abstract
Objective
Despite the increasing numbers of published trials of quality improvement (QI) interventions in diabetes, little is known about the risk of bias in this literature.
Design
Secondary analysis of a systematic review.
Data sources
Medline, the Cochrane Effective Practice and Organisation of Care (EPOC) database (from inception to July 2010) and references of included studies.
Eligibility criteria
Randomised trials assessing 11 predefined QI strategies or financial incentives targeting health systems, healthcare professionals or patients to improve the management of adult outpatients with diabetes.
Analysis
Risk of bias (low, unclear or high) was assessed for the 142 trials in the review across nine domains using the EPOC version of the Cochrane Risk of Bias Tool. We used Cochran-Armitage tests for trends to evaluate the improvement over time.
Results
There was no significant improvement over time in any of the risk of bias domains. Attrition bias (loss to follow-up) was the most common source of bias, with 24 trials (17%) having high risk of bias due to incomplete outcome data. Overall, 69 trials (49%) had at least one domain with high risk of bias. Inadequate reporting frequently hampered the risk of bias assessment: allocation sequence was unclear in 82 trials (58%) and allocation concealment was unclear in 78 trials (55%). There were significant reductions neither in the proportions of studies at high risk of bias over time nor in the adequacy of reporting of risk of bias domains.
Conclusions
Nearly half of the included QI trials in this review were judged to have high risk of bias. Such trials have serious limitations that put the findings in question and therefore inhibit evidence-based QI. There is a need to limit the potential for bias when conducting QI trials and improve the quality of reporting of QI trials so that stakeholders have adequate evidence for implementation.
Keywords: DIABETES & ENDOCRINOLOGY, STATISTICS & RESEARCH METHODS
Article summary.
Article focus
Reliable quality improvement research is needed to make decisions about initiating or scaling up quality improvement strategies.
The number of published quality improvement trials has increased rapidly over time.
The quality of trials published in other areas of health seem to be improving over time but the risk of bias in the quality improvement literature is uncertain.
Key messages
Nearly half of quality improvement trials for diabetes are at high risk of bias.
The quality of quality improvement trials does not seem to be improving over time.
Policy-makers, administrators, clinicians and research funders must carefully scrutinize the methods used in quality improvement trials to ensure evidence-based quality improvement.
Strengths and limitations of this study
This is the largest systematic review of risk of bias in the quality improvement literature and the only to assess for trends over time.
The risk of bias tool does not capture all sources of methodological bias and poor reporting interferes with the assessment of many domains.
The merits of any given trial report depend to some extent on the needs of the reader, such that some trials with high risk of bias may be of value for certain purposes.
Introduction
There is significant interest in quality improvement (QI) in healthcare, as evidenced by the rapidly increasing number of randomised clinical trials (RCTs) of QI interventions, especially in the diabetes literature.1 RCTs can provide a foundation for making statements regarding causation, but the validity of trials varies widely; trials with adequate allocation concealment and blinding generally produce smaller effect sizes.2 Since internal validity in QI trials is a necessary precursor for application to other settings,3 the ‘risk of bias’ of the findings should be assessed to ascertain the utility of the trial results. When an RCT is deemed to have high risk of bias, the study's findings become questionable.4
Evaluations to assess trends in methodological quality of RCTs have been conducted in many fields of healthcare,5 but no previous reviews have assessed risk of bias in QI RCTs or whether risk of bias in QI RCTs has changed over time. Recently, we conducted a systematic review and metaregression that included 142 RCTs evaluating QI strategies to improve care for patients with diabetes.1 In this secondary analysis of those data, we aimed to examine the risk of bias of included studies using the Cochrane Risk of Bias tool developed by the Cochrane Effective Practice and Organisation of Care (EPOC) group6 and determine whether the proportion with high risk of bias decreased over time. We also evaluated the trial and publication characteristics that might be associated with high risk of bias. Finally, we assessed whether the adequacy of reporting of risk of bias domains improved over time.
Methods
A detailed description of the methods used for searching, screening and abstracting the relevant data has been published1 and is briefly summarised here.
Search strategy
Studies were identified by searching MEDLINE and the Cochrane EPOC database (up to July 2010), and screening references of included RCTs. The search strategy has been previously published1 and is available on request.
Study selection
RCTs examining 1 of the 11 predefined QI strategies, and/or financial incentives, targeting health systems and/or healthcare professionals for the management of adult outpatients with diabetes were included. RCTs had to report at least one of the chosen process of care measures (proportion of patients taking acetylsalicylic acid, statins, antihypertensive medication, screened for retinopathy, screened for foot abnormalities and monitored for renal function) or intermediate outcomes (glycosylated haemoglobin levels, low-density lipoprotein cholesterol levels, diastolic and systolic blood pressure, proportion of patients with controlled hypertension and proportion of patients who quit smoking) for inclusion.
Data abstraction
A draft data abstraction form was developed and modified after a training exercise among reviewers. Two reviewers abstracted relevant data for each RCT independently. Discrepancies were resolved by discussion or the involvement of a third reviewer. Authors of the included RCTs were contacted to obtain further information for data items requiring clarification. Journal impact factors from journal citation reports (ISI Web of Science, 2009) were obtained. When a journal's ranking was unavailable, we used the impact ranking of the open access SMImago journal and country rank database if available.7 This ranking is calculated using a similar formula and is strongly correlated with the journal citation impact factor.8
Assessing risk of bias
As the included trials tested QI interventions, the Cochrane EPOC Risk of Bias Tool6 was used to assess the risk of bias in each study. The standard Cochrane Risk of Bias Tool includes an assessment of seven domains: sequence generation, allocation concealment, blinding of participants and personnel, blinding of outcome assessment, incomplete outcome data, selective reporting and others. The Cochrane Handbook9 provides instructions for making judgements about the specific domains as high, unclear or low risk. When formulating summary assessments for each trial, classification of a study as ‘high risk’ indicates that bias could have affected the results, while unclear risk of bias indicates that some doubt exists about the results, and low risk of bias indicates that bias is unlikely to affect the results. It has been shown empirically that studies classified as high risk using this tool are more likely to have larger effect sizes.10
The EPOC tool was adapted to account for the unique features of QI trials. (The guidelines for applying the Cochrane EPOC tool are summarised in table 1.) For example, in many QI trials, it is not possible to blind participants. In addition, QI trials may require cluster-randomisation to avoid contamination, but in cluster-randomised trials balance at baseline is a particular concern.11 Therefore, the EPOC tool uses the same approach as the general Cochrane Risk of Bias Tool, but requires an assessment of bias in nine domains: sequence generation, allocation concealment, similarity of baseline measurements, similarity of baseline characteristics, incomplete data, blinding of outcome assessment, contamination, selective outcome reporting and others. If a given domain is deemed ‘unclear’, it was inadequately reported to determine whether it meets high risk or low risk criteria. Risk of bias assessment was conducted independently by a clinician-researcher (NMI) and a systematic review methodologist (ACT), and conflicts were resolved by discussion with an expert QI trialist (JMG).
Table 1.
Cochrane Effective Practice and Organisation of Care risk of bias assessment tool*
| Risk of bias domain | Low risk of bias | High risk of bias | Unclear risk of bias |
|---|---|---|---|
| Was the allocation sequence adequately generated? | A random component in the sequence generation process is described (eg, referring to a random number table) | Non-random method is used (eg, performed by date of admission) | Not specified in the paper |
| Was the allocation adequately concealed? | The unit of allocation was by institution, team or professional and allocation was performed on all units at the start of the study; or if the unit of allocation was by patient or episode of care and there was some form of centralised randomisation scheme, an on-site computer system or sealed opaque envelopes were used | Allocation was not adequately concealed | Not specified in the paper |
| Were baseline outcome measurements similar? | Performance or patient outcomes were measured prior to the intervention, and no important differences were present across study groups, or if imbalanced but appropriate adjusted analysis was performed | Important differences were present and not adjusted for in analysis | If no baseline measure of outcome |
| Were baseline characteristics similar? | Baseline characteristics of the study and control providers are reported and similar | No report of characteristics in text or tables or if there are differences between control and intervention providers | Not clear in the paper |
| Were incomplete outcome data adequately addressed? | Missing outcome measures were unlikely to bias the results | Missing outcome data was likely to bias the results | Not specified in the paper |
| Was knowledge of the allocated interventions adequately prevented? | The authors state explicitly that the primary outcome variables were assessed blindly, or the outcomes are objective (eg, length of hospital stay) | Outcomes were not assessed blindly and not objective | Not specified in the paper |
| Was the study free from selective outcome reporting? | There is no evidence that outcomes were selectively reported | Some important outcomes are omitted from the results | Not specified in the paper |
| Was the study adequately protected against contamination? | Allocation was by community, institution or practice and it is unlikely that the control group received the intervention. | It is likely that the control group received the intervention. | Communication between intervention and control professionals could have occurred |
| Was the study free from other risks of bias? | There is no evidence of other risk of biases |
*Adapted for ease of presentation. See http://epoc.cochrane.org/epoc-resources-review-authors for full explanation.
Analysis
For each risk of bias domain, the proportions of RCTs meeting the criteria for high or low or unclear risk of bias were determined. To assess for trends over time in the bias classifications, year of publication was categorised into three groups demarcated by the publication of the 2001 CONSORT statement12 and the publication of the earlier version of the systematic review of diabetes QI interventions in 2006,13 as we believed that these may have spurred investigators to improve the quality of their trial. Therefore, we categorised the year of publication as before 2002; 2002–2006; and 2007–2010. We examined each of the risk of bias domains for change over time descriptively and conducted either exact or asymptotic Cochran-Armitage tests for trend for each item.
We estimated the proportion of QI RCTs at high risk of bias overall, together with 95% asymptotic CI. For this analysis, we created a dichotomous indicator for each RCT based on whether or not the study was classified as high risk of bias in at least one domain. To assess for trends in reporting over time, we dichotomised domains as ‘reported’ (low or high risk of bias) and ‘unreported’ (unclear risk of bias). We tested for trend over time in the proportion at high risk of bias overall, hypothesising that the proportion would decline over time. We used the same year of publication categories and conducted Cochran-Armitage tests for trend of the dichotomous indicator.
We also conducted a post hoc sensitivity analysis that applied an empirically based rule for assigning high risk of bias overall. Since previous meta-analyses have found that high risk of bias in four specific domains, namely allocation sequence generation, allocation concealment, blinding and selective outcome reporting are each associated with greater effect size,14–16 we repeated the analyses considering only studies with high risk of bias in these domains as high risk of bias overall.
Finally, we tested for associations between high risk of bias in at least one domain and study characteristics chosen a priori: type of diabetes (type 1, type 2, both or unclear), type of allocation (cluster randomised or patient randomised), country (USA or Canada, UK or Western Europe or others), type of intervention (single or multifaceted), journal impact factor, effective sample size and year of publication using χ2 tests (or Fisher's exact tests, as appropriate) for categorical and Wilcoxon signed-rank tests for continuous measures. We hypothesised that each of these characteristics may be associated with studies at high risk of bias overall.
All analyses were conducted in SAS V.9.2.17
Results
See figure 1 for a study flow diagram.
Figure 1.
Study flow diagram.
We analysed 142 studies, with 37 (26%) published before 2002, 46 (32%) between 2002 and 2006, and 59 (42%) between 2007 and 2010. These studies evaluated the effects of QI interventions on 123 529 patients with diabetes. Trial and patient characteristics are described in table 2. The proportions of studies judged to be at low, unclear or high risk of bias for each domain are illustrated in figure 2. The domains most commonly at high risk of bias were outcome reporting bias (17%) and similarity across characteristics at baseline (16%). A lack of similarity in outcome measures at baseline (10%), and lack of adequate blinding (8%) were also relatively common domains with high risk of bias. Studies were rarely at high risk of bias due to the allocation sequence generation (4%) or allocation concealment (3%), but these domains were often unclearly reported (57% and 55% unclear, respectively). Selective outcome reporting was deemed unclear approximately 84% of the time because published protocols were rarely available and it was often plausible that many more outcomes than those reported were measured. Table 3 indicates a lack of significant trend over time in the proportion of trials at high risk of bias for any given domain. Examination of table 3 also reveals no trends over time in quality of reporting for any of the risk of bias domains.
Table 2.
Study and patient characteristics
| Characteristic | Result |
|---|---|
| Patient RCTs, n (%) | 94 (66.2) |
| Cluster RCTs, n (%) | 48 (33.8) |
| Number of clusters, median (IQR) | 29 (12, 57) |
| Number of patients, median (IQR) | 405.3 (203, 878) |
| Duration of intervention months, median (IQR) | 12 (8.9, 15.0) |
| Mean age in years, median (IQR) | 59.4 (54.9, 62.9) |
| Percentage of male, median (IQR) | 49.8 (41.8, 55.9) |
| Type of diabetes n (%) | |
| Type 1 diabetes | 9 (6.3) |
| Type 2 diabetes | 80 (56.3) |
| Type 1 and 2 diabetes | 34 (23.9) |
| Type of diabetes unclear/NR | 19 (13.4) |
| Number of QIs per RCT median (IQR) | 2 (0, 3.5) |
| Administrators of patient intervention(s), n (%) | |
| Primary care physician | 30 (21.1) |
| Nurse | 67 (47.2) |
| Pharmacist | 19 (13.4) |
| Dietician | 22 (15.5) |
| Psychiatrist | 3 (2.1) |
| Psychologist | 2 (1.4) |
| Ophthalmologist | 2 (1.4) |
| Specialist/endocrinologist | 21 (14.8) |
| Other | 49 (34.5) |
| Location of study, n (%) | |
| The USA | 68 (47.9) |
| The UK | 14 (9.9) |
| Canada | 11 (7.7) |
| The Netherlands | 8 (5.6) |
| South Korea | 7 (4.9) |
| Australia | 6 (4.2) |
| Denmark | 3 (2.1) |
| Belgium | 1 (0.7) |
| Israel | 3 (2.1) |
| Spain | 3 (2.1) |
| Norway | 2 (1.4) |
| France | 2 (1.4) |
| Germany | 2 (1.4) |
| Italy | 2 (1.4) |
| Switzerland | 2 (1.4) |
| China | 2 (1.4) |
| Ireland | 1 (0.7) |
| New Zealand | 1 (0.7) |
| Thailand | 1 (0.7) |
| Taiwan | 1 (0.7) |
| United Arab Emirates | 1 (0.7) |
| Mexico | 1 (0.7) |
All IQRs reported as the 25th and 75th percentiles, includes investigators and community workers.
N, number; NA, not applicable; NR, not reported; QI, quality improvement; RCT, randomised clinical trial.
Figure 2.
Percentage of studies judged to be at low, unclear or high risk of bias in each risk of bias domain.
Table 3.
Trends over time in proportions of trials classified high, unclear or low for each risk of bias domain
| Risk of bias domain | Pre-2002 N=37 | 2002–2006 N=46 | 2007–2010 N=59 | p Value* | p Value† |
|---|---|---|---|---|---|
| Was the allocation sequence adequately generated? | 0.41 | 0.43 | |||
| Low | 11 (30%) | 19 (41%) | 25 (42%) | ||
| Unclear | 24 (65%) | 25 (55%) | 33 (56%) | ||
| High | 2 (5%) | 2 (4%) | 1 (2%) | ||
| Was the allocation adequately concealed? | 1.00 | 0.82 | |||
| Low | 15 (40%) | 20 (44%) | 25 (42%) | ||
| Unclear | 21 (57%) | 25 (54%) | 32 (54%) | ||
| High | 1 (3%) | 1 (2%) | 2 (4%) | ||
| Were baseline outcomes similar? | 0.87 | 0.20 | |||
| Low | 31 (84%) | 39 (85%) | 44 (75%) | ||
| Unclear | 2 (5%) | 3 (6%) | 8 (13%) | ||
| High | 4 (11%) | 4 (9%) | 7 (12%) | ||
| Were baseline characteristics similar? | 0.16 | 0.57 | |||
| Low | 30 (81%) | 34 (74%) | 43 (73%) | ||
| Unclear | 3 (8%) | 6 (13%) | 3 (5%) | ||
| High | 4 (11%) | 6 (13%) | 13 (22%) | ||
| Were incomplete outcome data adequately addressed? | 0.17 | 0.70 | |||
| Low | 29 (78%) | 33 (72%) | 38 (64%) | ||
| Unclear | 3 (8%) | 8 (17%) | 7 (12%) | ||
| High | 5 (14%) | 5 (11%) | 14 (24%) | ||
| Was knowledge of the allocated interventions prevented? | 0.44 | 0.61 | |||
| Low | 32 (87%) | 38 (83%) | 46 (78%) | ||
| Unclear | 3 (8%) | 5 (11%) | 7 (12%) | ||
| High | 2 (5%) | 3 (6%) | 6 (10%) | ||
| Was the study protected against contamination? | 0.54 | 0.78 | |||
| Low | 25 (68%) | 23 (50%) | 35 (59%) | ||
| Unclear | 10 (27%) | 21 (46%) | 19 (32%) | ||
| High | 2 (5%) | 2 (4%) | 5 (9%) | ||
| Was the study free from selective outcome reporting? | 0.84 | 1.00 | |||
| Low | 3 (8%) | 4 (9%) | 6 (10%) | ||
| Unclear | 32 (87%) | 37 (80%) | 50 (85%) | ||
| High | 2 (5%) | 5 (11%) | 3 (5%) | ||
| Was the study free from other risks of bias? | 0.58 | 0.58 | |||
| Low | 27 (73%) | 30 (65%) | 39 (66%) | ||
| Unclear | 10 (27%) | 16 (35%) | 20 (34%) | ||
| High | 0 | 0 | 0 |
*Exact Cochran-Armitage test for high versus low or unclear risk of bias in each domain except the last domain which was analysed as low versus high or unclear due to absence of studies with high risk of bias.
†Exact Cochran-Armitage test for reported (high or low risk of bias) or unreported (unclear risk of bias) in each domain.
Overall, 48.6% (69/142) of the RCTs had a high risk of bias in at least one domain (95% CI 40.4 to 56.8). Figure 3 illustrates the rapid increase in number of QI RCTs published over time and the cumulative proportion of trials having at least one domain with high risk of bias up to a given year. In general, the line representing the proportion at high risk of bias runs parallel to the number of trials published, consistently accounting for almost half of the studies. Table 4 indicates a lack of significant trend over time in the proportion of trials with at least one domain with high risk of bias: these proportions were 46%, 44% and 54% before 2002, between 2002 and 2006, and after 2006, respectively. Table 4 also demonstrates a lack of significant association between any of the study characteristics considered and the presence of high risk of bias in at least one domain.
Figure 3.
Cumulative number of diabetes quality improvement trial publications at high risk of bias in any domain, 1990–2010.
Table 4.
Association between study characteristics and risk of bias
| Characteristic | All studies (n) | Studies in high risk of bias in at least one domain number (%) | p Value* |
|---|---|---|---|
| Year of publication | 0.37 | ||
| Pre-2002 | 37 | 17 (46%) | |
| 2002–2006 | 46 | 20 (44%) | |
| 2007–2010 | 59 | 32 (54%) | |
| Type of diabetes | 0.11 | ||
| Type 1 | 9 | 3 (33%) | |
| Type 2 | 80 | 36 (45%) | |
| Both | 34 | 16 (47%) | |
| Unclear | 19 | 14 (74%) | |
| Unit of allocation | 0.24 | ||
| Patient | 94 | 49 (52%) | |
| Cluster (eg, provider/clinic) | 48 | 20 (42%) | |
| Country/setting | 0.62 | ||
| The USA or Canada | 79 | 41 (52%) | |
| The UK or Western Europe | 40 | 17 (43%) | |
| Other | 23 | 11 (48%) | |
| Journal impact factor | 0.87 | ||
| Greater than 3 (median) | 71 | 34 (47.9%) | |
| Less than 3 (median) | 71 | 35 (49.3%) | |
| Effective sample size | 0.87 | ||
| Greater than 154 (median) | 71 | 35 (49.3%) | |
| Less than 154 (median) | 71 | 34 (47.9%) | |
| Intervention type | 0.17 | ||
| Multifaceted (featuring more than one QI strategy) | 124 | 63 (51%) | |
| Single intervention | 18 | 6 (33%) |
*Comparing proportion of studies with at least one domain at high risk of bias against studies no domains at high risk of bias. For year of publication, Cochran-Armitage test for trend was conducted. For other study characteristics, χ2 (or Fisher's exact) tests for categorical and Wilcoxon signed-rank tests for continuous variables were used.
QI, quality improvement.
The sensitivity analysis, restricting studies defined as high risk of bias overall to those with high risk of bias in one of four domains (allocation sequence generation, allocation concealment, blinding or selective outcome reporting) also revealed no trends over time—the proportions were 19%, 20% and 20% before 2002, between 2002 and 2006 and after 2006, respectively (p=0.86).
Discussion
Main findings
Using the Cochrane EPOC Risk of Bias Tool,6 we found that nearly half of RCTs focusing on diabetes had at least one domain at high risk of bias. The trials were most often at high risk of bias due to inadequate follow-up of participants, a lack of similarity at baseline across outcome measures or covariates, or inadequate blinding. We also noted that the majority of RCT reports failed to include an adequate description of the allocation process (ie, sequence generation and allocation concealment were ‘unclear’). To be interpreted appropriately, RCTs must be completely and transparently reported.18 19 Our findings indicate that greater efforts are needed to ensure both adequate reporting and methodological conduct of diabetes QI trials.
We found that poor follow-up, baseline imbalances and blinding were the most common sources of high risk of bias. Although these domains may be difficult fully control in QI trials, methodological approaches are available to mitigate and/or explore such causes of risk of bias. For example, sensitivity analyses may be used to explore the risk of bias related to loss of follow-up, and risk of baseline imbalances in QI trials may be reduced through restricted randomisation techniques, especially when trials are cluster-randomised with relatively few clusters. In addition, selective outcome reporting may be limited if more QI trial protocols were registered. Finally, although blinding may be particularly difficult to accomplish in QI trials, this should be clearly reported; if outcome assessment is not blinded, risk of bias could still be limited by using objective outcomes.
Comparison to literature
A systematic review focusing on cluster randomised trials found minimal improvement over time in either reporting or methodological conduct.20 We found no evidence for a difference in the proportion of cluster-randomised trials at high risk of bias compared with trials in which individuals were allocated. However, imbalance at baseline was a common source of potential bias in diabetes QI trials, possibly owing to the inadequate use of restricted randomisation in cluster trials.21 Another systematic review included 35 studies covering a range of health-related fields assessing trends over time in quality criteria for RCTs.5 Of these, 26 found improvement over time for at least one aspect of methodological quality. The domain most commonly noted to have improvement was allocation concealment, but the authors noted that this domain remained either poorly reported or inadequately performed in over half of the examined trials. We found a similarly low proportion of studies clearly reporting adequate allocation concealment, and no evidence of improvement over time.
Previous authors have noted that QI reports may not contain enough information to inform generalisation and allow for replication in different clinical settings.22 Standards for Quality Improvement Reporting (SQUIRE) guidelines suggest that investigators conducting trials use both SQUIRE and CONSORT to inform their manuscripts.19 Journal editors should enforce the requirements of both SQUIRE and CONSORT for QI RCTs, possibly by permitting detailed information to be posted as online appendices. Although it might seem onerous to force investigators to address all items in SQUIRE and CONSORT, the risks of poor reporting are substantial. Inadequate description of context could omit essential preconditions or important effect modifiers for a successful QI programme, while incomplete description of the programme itself might lead to failure due to partial implementation.
Strengths and limitations
To our knowledge, this is the largest analysis of risk of bias ever reported for healthcare QI RCTs and the only one to assess for trends over time. The findings are strengthened by the rigorous methods used to prepare the data for the systematic review.
QI evaluations have been criticised based on numerous criteria beyond the risk of bias domains, including short duration of intervention, lack of justification for intervention design and poor generalisability.23 24 Some important components of methodological quality do not relate to bias (eg, reporting of a sample size calculation). Thus, it is possible that studies at low risk of bias have important flaws with respect to methodology and/or reporting (and vice versa), and it is possible that using other scales to assess study quality could have led to different results.14 While the overall risk of bias assessment using the Cochrane Risk of Bias Tool has been shown to differentiate effect sizes (ie, higher risk of bias studies usually have larger effect sizes),10 studies at high risk of bias may still offer valuable knowledge for QI implementers. The merit of any given report will depend on the needs of the reader, while the current analysis provides an assessment of the progress in the literature as a whole.
Furthermore, we acknowledge that assigning trials with high risk of bias in a single domain a status of high risk of bias overall may be arguable. Nevertheless, our sensitivity analysis led to the same conclusion: there has been no improvement over time in the proportion of trials at high risk of bias in this literature and no particular study characteristics were associated with high risk of bias.
Another potential limitation stems from our analytical approach regarding change over time; collapsing publication year into three timeframes (pre-2002, 2002–2006 and 2007–2010) and testing for trends may have limited our power. These timeframes were chosen a priori based on the publication of important documents that we thought might affect the conduct and reporting of these trials. We felt the assumption of linear change over time underlying the Cochran-Armitage test for trend was appropriate and in keeping with our hypotheses (eg, high and unclear risk of bias would decrease gradually over time, while low risk of bias would increase). Risk of type 2 error is tempered by the number of tests performed; the lack of a significant p value for trend for any level of risk of bias in any domain supports our main conclusion. Finally, this review considered only RCTs from the diabetes literature. It would have been preferable to evaluate a random sample of all QI trials, but adequate QI electronic literature searches are yet to be developed.25
Implications
Published trials testing QI in diabetes are frequently at high risk of bias, producing results that may not be replicable. Clinicians must scrutinise the internal validity of the results as a first step in the process of considering the application of clinical findings for particular patients. Our findings emphasise the need for policy-makers, managers and/or clinical-administrators seeking to implement QI interventions to apply the same process.3 It is likely that QI investigators publishing RCTs desire for their work to have a broad impact. To help them in accomplishing this, research funders and journal editors can play an important role by ensuring that QI trials are reported thoroughly and transparently and are designed in a manner that limits the potential for risk of bias.
Supplementary Material
Acknowledgments
We would like to thank Jennifer D'Souza for her help in formatting the manuscript.
Footnotes
Contributors: NMI and ACT designed and coordinated the study, participated in data collection, data analysis, data interpretation and drafted the manuscript. MT conducted the analysis and participated in data interpretation and drafting the manuscript. IH, LT, DM and JMG helped to design the study and write the manuscript. All authors read and approved the final manuscript.
Funding: Ontario Ministry of Health and Long-Term Care and the Alberta Heritage Fund supported the initial systematic review.
Competing interests: This research received no specific grant from any funding agency in the public, commercial or not-for-profit sectors. The authors declare that (1) NMI, ACT, MT, IH, LT, DM and JMG received support from Ontario Ministry of Health and Long-term Care and the Alberta Heritage Foundation for the original systematic review, but the funding agencies had no role in the study design, collection, analysis or interpretation of data, writing of the manuscript or in the decision to submit this manuscript for publication; (2) NMI, ACT, MT, IH, LT, DM and JMG have no relationships with any companies that might have an interest in the submitted work in the previous 3 years; (3) their spouses, partners or children have no financial relationships that may be relevant to the submitted work and (4) NMI, ACT, MT, IH, LT, DM and JMG have no non-financial interests that may be relevant to the submitted work. NMI holds fellowship awards from the Canadian Institutes of Health Research (CIHR) and from the Department of Family and Community Medicine, University of Toronto. ACT holds a CIHR/Drug Safety and Effectiveness Network New Investigator Award in Knowledge Synthesis. DM holds a University Research Chair at the University of Ottawa and JMG holds a Canada Research Chair in Health Knowledge Transfer and Uptake.
Provenance and peer review: Not commissioned; externally peer reviewed.
Data sharing statement: Data detailing the risk of bias for each of the 142 trials in the review are available upon request.
References
- 1.Tricco AC, Ivers NM, Grimshaw JM, et al. Effectiveness of quality improvement strategies on the management of diabetes: a systematic review and meta-analysis. Lancet 2012;379:2252–61 [DOI] [PubMed] [Google Scholar]
- 2.Pildal J, Hrobjartsson A, Jorgensen KJ, et al. Impact of allocation concealment on conclusions drawn from meta-analyses of randomized trials. Int J Epidemiol 2007;36:847–57 [DOI] [PubMed] [Google Scholar]
- 3.Fan E, Laupacis A, Pronovost PJ, et al. How to use an article about quality improvement. JAMA 2010;304:2279–87 [DOI] [PubMed] [Google Scholar]
- 4.Higgins JP, Altman DG, Gotzsche PC, et al. The Cochrane Collaboration's tool for assessing risk of bias in randomised trials. BMJ 2011;343:d5928. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Falagas ME, Grigori T, Ioannidou E. A systematic review of trends in the methodological quality of randomized controlled trials in various research fields. J Clin Epidemiol 2009;62:227–31, 31 e1–9 [DOI] [PubMed] [Google Scholar]
- 6.Cochrane Effective Practice and Organisation of Care Group Suggested risk of bias criteria for EPOC reviews, 2012
- 7. SJR—SCImago Journal & Country Rank, 2007 [program]
- 8.Falagas ME, Kouranos VD, Arencibia-Jorge R, et al. Comparison of SCImago journal rank indicator with journal impact factor. FASEB J 2008;22:2623–8 [DOI] [PubMed] [Google Scholar]
- 9.Higgins JPT, Altman DG. Assessing risk of bias in included studies. In: Higgins JPT, Green S. (editors). Cochrane Handbook for Systematic Reviews of Interventions Version 5.1.0 [updated March 2011]. The Cochrane Collaboration, 2011. Available from www.cochrane-handbook.org.
- 10.Hartling L, Ospina M, Liang Y, et al. Risk of bias versus quality assessment of randomised controlled trials: cross sectional study. BMJ 2009;339:b4012. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11.Ivers NM, Halperin IJ, Barnsley J, et al. Allocation techniques for balance at baseline in cluster randomized trials: a methodological review. Trials 2012;13:120. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 12.Moher D, Schulz KF, Altman D, et al. The CONSORT statement: revised recommendations for improving the quality of reports of parallel-group randomized trials. JAMA 2001;285:1987–91 [DOI] [PubMed] [Google Scholar]
- 13.Shojania KG, Ranji SR, McDonald KM, et al. Effects of quality improvement strategies for type 2 diabetes on glycemic control: a meta-regression analysis. JAMA 2006;296:427–40 [DOI] [PubMed] [Google Scholar]
- 14.Armijo-Olivo S, Stiles CR, Hagen NA, et al. Assessment of study quality for systematic reviews: a comparison of the Cochrane Collaboration Risk of Bias Tool and the Effective Public Health Practice Project Quality Assessment Tool: methodological research. J Eval Clin Pract 2012;18:12–18 [DOI] [PubMed] [Google Scholar]
- 15.Odgaard-Jensen J, Vist GE, Timmer A, et al. Randomisation to protect against selection bias in healthcare trials. Cochrane Database of Systematic Reviews 2011;4: MR000012. doi: 10.1002/14651858.MR000012.pub3. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.Hempel S, Suttorp MJ, Miles JNV, et al. Empirical Evidence of Associations Between Trial Quality and Effect Size [Internet]. Rockville, MD: Agency for Healthcare Research and Quality (US); 2011 [PubMed] [Google Scholar]
- 17.SAS 9.2 [program]: Cary, NC: SAS Institute Inc., 2008
- 18.Simera I, Moher D, Hirst A, et al. Transparent and accurate reporting increases reliability, utility, and impact of your research: reporting guidelines and the EQUATOR network. BMC Med 2010;8:24. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Davidoff F, Batalden P, Stevens D, et al. Publication guidelines for quality improvement studies in health care: evolution of the SQUIRE project. BMJ 2009;338:a3152. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Ivers NM, Taljaard M, Dixon S, et al. Impact of CONSORT extension for cluster randomised trials on quality of reporting and study methodology: review of random sample of 300 trials, 2000–8. BMJ 2011;343:d5886. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 21.Campbell MK, Elbourne DR, Altman DG, Group C . CONSORT statement: extension to cluster randomised trials. BMJ 2004;328:702–8 [DOI] [PMC free article] [PubMed] [Google Scholar]
- 22.Michie S, Fixsen D, Grimshaw JM, et al. Specifying and reporting complex behaviour change interventions: the need for a scientific method. Implement Sci 2009;4:40. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 23.Alexander JA, Hearld LR. What can we learn from quality improvement research? A critical review of research methods. Med Care Res Rev 2009;66:235–71 [DOI] [PubMed] [Google Scholar]
- 24.Eccles M, Grimshaw J, Walker A, et al. Changing the behavior of healthcare professionals: the use of theory in promoting the uptake of research findings. J Clin Epidemiol 2005;58:107–12 [DOI] [PubMed] [Google Scholar]
- 25.Hempel S, Rubenstein LV, Shanman RM, et al. Identifying quality improvement intervention publications—a comparison of electronic search strategies. Implement Sci 2011;6:85. [DOI] [PMC free article] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.