Risk of bias for studies with a separate control group (RCTs, NRCTs, CBAs)

Nine standard criteria are used for all RCTs, NRCTs and CBAs. Further information can be obtained from the Cochrane Handbook for Systematic Reviews of Interventions section on risk of bias and from the draft methods paper on risk of bias under the EPOC‐specific resources section of the EPOC website. Was the allocation sequence adequately generated? Score: “low risk” if a random component in the sequence generation process is described (e.g. referring to a random number table). Score “high risk” when a non‐random method is used (e.g. performed by date of admission). NRCTs and CBA studies should be scored “high risk”. Score “unclear risk” if not specified in the paper. Was the allocation adequately concealed? Score “low risk” if the unit of allocation was by institution, team or professional and allocation was performed on all units at the start of the study; or if the unit of allocation was by patient or episode of care, and some form of centralised randomisation scheme, an on‐site computer system or sealed opaque envelopes were used. CBA studies should be scored “high risk”. Score “unclear risk” if not specified in the paper. Were baseline outcome measurements similar?1,2 Score “Low risk” if performance or patient outcomes were measured prior to the intervention, and no important differences were present across study groups. In RCTs, score “Low risk” if imbalanced but appropriate adjusted analysis was performed (e.g. Analysis of covariance). Score “High risk” if important differences were present and not adjusted for in analysis. If RCTs have no baseline measure of outcome, score “Unclear risk”. Were baseline characteristics similar? Score “Low risk” if baseline characteristics of the study and control providers are reported and similar. Score “Unclear risk” if it is not clear in the paper (e.g. characteristics are mentioned in text but no data were presented). Score “High risk” if there is no report of characteristics in text or tables or if there are differences between control and intervention providers. Note that in some cases imbalance in patient characteristics may be due to recruitment bias whereby the provider was responsible for recruiting patients into the trial. Were incomplete outcome data adequately addressed?1 Score “low risk” if missing outcome measures were unlikely to bias the results (e.g. the proportion of missing data was similar in the intervention and control groups, or the proportion of missing data was less than the effect size, i.e. unlikely to overturn the study result). Score “high risk” if missing outcome data were likely to bias the results. Score “unclear risk” if not specified in the paper (do not assume 100% follow‐up unless stated explicitly). Was knowledge of the allocated interventions adequately prevented during the study?1 Score “low risk” if study authors state explicitly that the primary outcome variables were assessed blindly, or if the outcomes are objective (e.g. length of hospital stay). Primary outcomes are those variables that correspond to the primary hypothesis or question as defined by study authors. Score “high risk” if the outcomes were not assessed blindly. Score “unclear risk” if this is not specified in the paper. Was the study adequately protected against contamination? Score “low risk” if allocation was by community, institution or practice, and it is unlikely that the control group received the intervention. Score “high risk” if it is likely that the control group received the intervention (e.g. if patients rather than professionals were randomly assigned). Score “unclear risk” if professionals were allocated within a clinic or practice, and it is possible that communication between intervention and control professionals could have occurred (e.g. physicians within practices were allocated to intervention or control). Was the study free of selective outcome reporting? Score “low risk” if no evidence suggests that outcomes were selectively reported (e.g. all relevant outcomes in the Methods section are reported in the Results section). Score “high risk” if some important outcomes are subsequently omitted from the results. Score “unclear risk” if not specified in the paper. Was the study free of other risks of bias? Score “low risk” if no evidence suggests other risks of bias. _________________ 1If some primary outcomes were imbalanced at baseline, assessed blindly or affected by missing data and others were not, each primary outcome can be scored separately. 2If “unclear” or “no”, but sufficient data are provided in the paper for an adjusted analysis (e.g. baseline adjustment analysis, intention‐to‐treat analysis), the criteria should be rescored to “yes”.

Risk of bias for interrupted time series (ITS) studies

Seven standard criteria are used for all ITS studies. Further information can be obtained from theCochrane Handbook on Systematic Reviews of Interventions section on risk of bias and from the draft methods paper on risk of bias under the EPOC specific resources section of the EPOC website. Note: If the ITS study has ignored secular (trend) changes and performed a simple t‐test of before versus after intervention periods without further justification, the study should not be included in the review unless reanalysis is possible. Was the intervention independent of other changes? Score “low risk” if compelling arguments indicate that the intervention occurred independently of other changes over time, and that the outcome was not influenced by other confounding variables/historic events during the study period. If events/variables were identified, note what they are. Score “high risk” if it is reported that the intervention was not independent of other changes in time. Was the shape of the intervention effect prespecified? Score ”low risk” if point of analysis is the point of intervention OR a rational explanation for the shape of the intervention effect was given by the study author(s). When appropriate, this should include an explanation if the point of analysis is NOT the point of intervention; score “high risk” if it is clear that the condition above is not met. Was the intervention unlikely to affect data collection? Score “low risk” if it is reported that the intervention itself was unlikely to affect data collection (e.g. sources and methods of data collection were the same before and after the intervention); score “high risk” if the intervention itself was likely to affect data collection (e.g. any change in source or method of data collection reported). Was knowledge of the allocated interventions adequately prevented during the study?*** Score “low risk” if study authors state explicitly that the primary outcome variables were assessed blindly, or if the outcomes are objective (e.g. length of hospital stay). Primary outcomes are those variables that correspond to the primary hypothesis or question as defined by study authors. Score “high risk” if the outcomes were not assessed blindly. Score “unclear risk” if this is not specified in the paper. Were incomplete outcome data adequately addressed?*** Score “low risk” if missing outcome measures were unlikely to bias the results (e.g. the proportion of missing data was similar in the before‐ and after‐intervention periods, or if the proportion of missing data was less than the effect size (i.e. unlikely to overturn the study result). Score “high risk” if missing outcome data were likely to bias the results. Score “unclear risk” if this was not specified in the paper. (Do not assume 100% follow‐up unless this was stated explicitly.) Was the study free of selective outcome reporting? Score “low risk” if no evidence suggests that outcomes were selectively reported (e.g. all relevant outcomes in the Methods section were reported in the Results section). Score “high risk” if some important outcomes are subsequently omitted from the results. Score “unclear risk” if this was not specified in the paper. Was the study free of other risks of bias? Score “low risk” if no evidence suggests other risks of bias (e.g. should consider if seasonality is an issue, i.e. if January to June constitutes the preintervention period, and July to December the post, could the “seasons’ have caused a spurious effect?). ***If some primary outcomes were assessed blindly or were affected by missing data and others were not, each primary outcome can be scored separately.