Table 5.
Risk of bias for non-randomised studies using the Quality Assessment Tool for Quantitative Studies (QATSQ) tool
| Study | Selection bias | Study design | Confounders | Blinding | Data collection methods | Withdrawals and dropouts | Summary |
|---|---|---|---|---|---|---|---|
| Bapat et al. (2009) [59] | 2 | 2 | 3 | 3 | 3 | 3 | |
| Participants are very likely to be representative Cannot tell the percentage of participants who agreed | Study is designated as a cohort analytic study | There were gender and age differences that may have influenced the outcomes between participants and these were not controlled for in analysis | Outcome assessors knew intervention status, and blinding of participants to research question is not described | The validity and reliability of the instruments are not described | Withdrawals and dropouts were not described | Weak quality: as this study scored four weak ratings, the overall judgement is weak quality | |
| Breslin et al. (2017) [8, 60] | 2 | 1 | 1 | 2 | 1 | 3 | |
| Participants are very likely to be representative Cannot tell percentage of participants who agreed | Study is designated as a controlled clinical trial | Confounders (gender, sport type) were similar across control and intervention groups | Cannot tell if outcome assessors were aware of intervention status and cannot tell if intervention participants were aware of research question | Tools were shown to be valid and reliable | Cannot tell if there were withdrawals or dropouts | Moderate quality: As this study scored one weak rating the overall judgement is moderate quality | |
| Breslin et al. (2018) | 1 | 1 | 2 | 2 | 1 | 3 | |
| Participants are very likely to be representative. All participants agreed to participate | Study is designated as a controlled clinical trial | Age differences between groups may have acted as confounder. Other significant demographic differences were controlled for | Cannot tell if outcome assessors were aware of intervention status and cannot tell if intervention participants were aware of research question | Tools were shown to be valid and reliable | Significant drop out rate was described and reasons provided | Moderate quality; As this study scored one weak rating the overall judgement is moderate quality | |
| Chow et al. (2020) [46] | 1 | 2 | 2 | 3 | 1 | 1 | |
| Participants are very likely to be representative. All participants agreed to participate | Study is designated as a cohort analytic study | No significant baseline differences between those who had mental health experience and those who had not therefore groups were combined for primary analysis | Outcome assessors knew intervention status and blinding of participants to research question is not described | Tools were shown to be valid and reliable | 100% completion rate at follow-up | Moderate quality; as this study scored one weak rating the overall judgement is moderate quality | |
| Donohue et al. (2015) [62] | 1 | 2 | 3 | 3 | 1 | 2 | |
| Participants are very likely to be representative All participants agreed to participate | Study is designated as a cohort analytic study | There were gender, ethnic and age differences that may have influenced the direction of result. These were not controlled for in the analysis | Outcome assessors knew intervention status, and the participants knew intended outcome of the research (i.e. developing intervention) | The validity and reliability of the instruments is described | There was a 70% follow-up rate from those that consented and completed the intervention | Weak quality: as this study scored two weak ratings, the overall judgement is weak quality | |
| Dowell et al. (2020) [48] | 2 | 2 | 3 | 3 | 2 | 3 | |
| Participants are somewhat likely to be representative, fee required may influence sample. All participants agreed to participate | Study is designated as a cohort analytic study | The requirement to control confounders was alluded to but the rationale behind adjustment was not sufficiently transparent | Outcome assessors knew intervention status and blinding of participants to research question is not described | Some tools were shown to be valid and reliable, low internal consistency was observed for measuring conduct problems | Less than 50% of initial sample completed intervention | Weak quality; as this study scored three weak ratings the overall judgement is weak quality | |
| Dubuc-Charbonneau and Durand-Bush (2015) [49] | 1 | 2 | 3 | 3 | 1 | 1 | |
| Participants are very likely to be representative. All participants agreed to participate | Study is designated as a cohort analytic study | Confounding variables were not discussed | Outcome assessors knew intervention status and blinding of participants to research question is not described | Tools were shown to be valid and reliable | 100% completion rate at follow-up | Weak quality; as this study scored two weak ratings, the overall judgement is weak quality | |
| Fogaca (2019) [50] | 2 | 1 | 3 | 3 | 1 | 2 | |
| Participants are somewhat likely to be representative. Risk of selection bias by removal of one team from intervention group data. Above 80% of participants agreed to participate | Study is designated as a controlled clinical trial | Study showed that there were no significant differences between intervention and control for mental health measures pre-test with the exception of depression, as a result the outlying team was removed from the data. No discussion of demographic differences (potential confounders) between intervention and control | Outcome assessor knew intervention status and blinding of participants to research question is not described | Tools were shown to be valid and reliable | 60–79% completion rate | Weak quality; as this study scored two weak ratings, the overall judgement is weak quality | |
| Hurley et al. (2018) [18] | 1 | 1 | 2 | 3 | 1 | 1 | |
| Participants are very likely to be representative. All participants agreed to participate | Study is designated as a controlled clinical trial | The study deploys a ‘matched’ control group to attempt to control for confounding variables but no mention of whether this holds true | Outcome assessors knew intervention status and blinding of participants to research question is not described | Tools were shown to be valid and reliable | > 80% completion rate at follow-up | Moderate quality; as this study scored one weak rating, the overall judgement is moderate quality | |
| Hurley et al. (2020) [52] | 1 | 1 | 1 | 2 | 1 | 3 | |
| Participants are very likely to be representative. All participants agreed to participate | Study is designated as a controlled clinical trial | Matched control trial to account for confounding variables. Covariates are adjusted for | Cannot tell if outcome assessors were aware of intervention status and blinding of participants to research question is not described | Tools were shown to be valid and reliable | Retention of participants was low particularly in the control group | Moderate quality; as this study scored one weak rating, the overall judgement is moderate quality | |
| Laureano et al. (2014) [53] | 2 | 1 | 2 | 3 | 2 | 1 | |
| Participants are somewhat likely to be representative. All participants agreed to participate | Study is designated as a controlled clinical trial | Study corrected for pre-test differences. However, extraneous variables impacting cannot be ruled out | Outcome assessors knew intervention status and blinding of participants to research question is not described | Some tools were shown to be valid and reliable, FORQ results should be treated tentatively due to low internal consistency | 100% completion rate at follow-up of intervention and control groups | Moderate quality; as this study scored one weak rating, the overall judgement is moderate quality | |
| Longshore and Sachs (2015) [64] | 1 | 1 | 1 | 3 | 3 | 1 | |
| Participants are very likely to be representative Above 80% of participants agreed to participate | Study is designated as a controlled clinical trial. | No significant differences were found between the groups before the intervention | Outcome assessors knew intervention status, and the participants knew intended outcome of the research (i.e. benefits of mindfulness) | The validity and reliability of the instruments is not described | There was a > 80% follow-up rate from those that consented and completed the intervention | Weak quality: as this study scored two weak ratings, the overall judgement is weak quality | |
| Mohammed et al. (2018) [54] | 2 | 1 | 1 | 3 | 2 | 1 | |
| Participants are somewhat likely to be representative. >80% of participants agreed to participate | Study is designated as a controlled clinical trial | Confounders were similar across intervention and control group | Outcome assessors knew intervention status and participants were not blinded to research question | The tools deployed displayed varied levels of validity and reliability | > 80% completion rate at follow-up | Moderate quality; as this study scored one weak rating, the overall judgement is moderate quality | |
| Pierce et al. (2010) [65] | 2 | 2 | 3 | 3 | 3 | 2 | |
| Participants are very likely to be representative Cannot tell the percentage of participants who agreed | Study is designated as a cohort analytic study | There were age and education differences that may have influenced the direction of result these were not controlled for in the analysis | Outcome assessors knew intervention status, and the participants knew intended outcome of the research (i.e. respond to mental health problems) | The validity and reliability of the instruments is not described | There was a 66% follow-up rate from those that consented and completed the intervention | Weak quality: as this study scored three weak ratings, the overall judgement is weak quality | |
| Sebbens et al. (2016) [66] | 1 | 1 | 1 | 3 | 3 | 1 | |
| Participants are very likely to be representative Above 80% of participants agreed to participate | Study is designated as a controlled clinical trial | No significant demographic differences were found between the groups before the intervention | Outcome assessors knew intervention status, and blinding of participants to research question is not described | The validity and reliability of the instruments is not described | There was a > 80% follow-up rate from those that consented and completed the intervention | Weak quality: As this study scored two weak ratings, the overall judgement is weak quality | |
| Slack et al. (2015) [67] | 1 | 2 | 3 | 3 | 3 | 1 | |
|
Participants are very likely to be representative Above 80% of participants agreed to participate |
Study is designated as a cohort analytic study | Confounding variables were not discussed | Outcome assessors knew intervention status, and blinding of participants to research question is not described | While one measure was referenced as valid and reliable, no information was reported on validity and reliability of another measure (RSMT) | There was a > 80% follow-up rate from those that consented and completed the intervention | Weak quality: As this study scored three weak ratings, the overall judgement is weak quality | |
| Shannon et al. (2019) [56] | 1 | 1 | 1 | 3 | 1 | 2 | |
| Participants are very likely to be representative. All participants agreed to participate | Study is designated as a controlled clinical trial | Baseline measurements indicated that there were no significant differences between control and intervention group for study outcomes or gender. Age was significantly different but analysis showed it did not have a significant effect on outcomes | Outcome assessors knew intervention status and blinding of participants to research question is not described | Tools were shown to be valid and reliable | There is no information provided about withdrawals or dropouts but Little’s MCAR analyses revealed data was missing at random and the expectation maximisation algorithm was used to estimate missing values | Moderate quality; as this study scored one weak rating, the overall judgement is moderate quality | |
| Tester et al. (1999) [68] | 2 | 2 | 3 | 2 | 1 | 3 | |
| Participants are very likely to be representative Cannot tell the percentage of participants who agreed | Study is designated as a cohort analytic study | Confounding variables were not discussed | Cannot tell if outcome assessors were aware of intervention status Cannot tell if intervention participants were aware of research question | Tools were referenced as valid and reliable | Cannot tell if there were withdrawals or dropouts | Weak quality: As this study scored two weak ratings, the overall judgement is weak quality | |
| Vella et al. (2020) [57] | 1 | 1 | 1 | 2 | 2 | 3 | |
| Participants are very likely to be representative. All participants agreed to participate | Study is designated as a controlled clinical trial | Matched control to account for confounding variables. Baseline differences are highlighted and adjusted for | Cannot tell if outcome assessors were aware of intervention status and blinding of participants to research question is not described | Majority of tools were shown to be valid and reliable except low internal consistency for implicit beliefs scale | A small proportion of participants completed the entire intervention per protocol | Moderate quality; as this study scored one weak rating, the overall judgement is moderate quality | |
| Vidic et al. (2018) [58] | 1 | 2 | 3 | 3 | 1 | 3 | |
| Participants are very likely to be representative. All participants agreed to participate | Study is a cohort design | Did not control for confounding variables | Outcome assessor knew intervention status and blinding of participants to research question is not described | Tool used was shown to be valid and reliable | There is no information provided about withdrawals or dropouts | Weak quality; as this study scored more than two weak ratings, the overall judgement is weak quality | |
| Summary of bias across the studies | Twelve studies were of strong quality and controlled for selection bias, the remaining 8 were of moderate quality | Eleven studies were of strong quality for study design and the remaining 9 were of moderate quality | There was a mixture of strong (n = 7), weak (n = 9) and moderate (n = 4) information provided on confounders | Fifteen of the non-randomised studies were of weak quality for blinding participants and outcome assessors. 5 were of moderate quality | Eleven of the non-randomised studies were of strong quality and referenced adequate reliability and validity for outcome measures, while 9 studies used tools of varied validity | There was a mixture of strong (n = 8), weak (n = 8) and moderate (n = 4) for the researchers’ disclosure of follow-up rates and dropouts | Nine studies were deemed to be of moderate quality and 11 were of weak quality |
1 = strong; 2 = moderate; 3 = weak