Table 3.
Quality assessment: Mixed Methods Appraisal Tool (MMAT), version 2018 [39]
| 1. Category: Qualitative study | Quality assessment | ||||||||
| Authora, year | Are there clear research questions? | Do the collected data allow to address the research questions? | Is the qualitative approach appropriate to answer the research question? | Are the qualitative data collection methods adequate to address the research question? | Are the findings adequately derived from the data? | Is the interpretation of results sufficiently substantiated by data? | Is there coherence between qualitative data sources, collection, analysis and interpretation? | Total MMAT score | Percent agreement of authors |
| Aborg [4] | 1 | 1 | 1 | 1 | 1 | 0 | 1 | 6 | 100% |
| (p. 6) | (p. 8) | (p. 15) | (p. 8) | (p. 8) | the sample size was gradually reduced | (p. 19) | |||
| Jaakson [2] | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 4 | 86% |
| (p. 198) | (p. 200, 201) | (p. 203) | (p. 202, 203) | not reported | unclear | not reported | |||
| Olsen [40] | 1 | 1 | 1 | 1 | 1 | 1 | 1 | 7 | 100% |
| (p. 345) | (p. 346-347) | (p. 345) | (p. 345) | (p. 345, 346) | (p. 346f.) | (p. 345) | |||
| Montreuil [28] | 1 | 1 | 1 | 1 | 0 | 1 | 0 | 5 | 100% |
| (p. 341) | (p. 341) | (p. 341) | (p. 341) | unclear | (p. 343f.) | unclear | |||
| (p. 343) | |||||||||
| 2. Category: Quantitative randomized controlled trials | Quality assessment | ||||||||
| Authora, year | Are there clear research questions? | Do the collected data allow to address the research questions? | Is randomization appropriately performed? | Are the groups comparable at baseline? | Are there complete outcome data? | Are outcome assessors blinded to the intervention provided? | Did the participants adhere to the assigned intervention? | Total MMAT Score | Percent agreement of authors |
| Harrington [20] | 1 | 1 | 0 | 1 | 1 | 0 | 0 | 4 | 86% |
| (p. 15) | (p. 15) | unclear | (p. 18) | (p. 17, 18) | not reported | not reported | |||
| (p. 17) | (p. 16) | ||||||||
| 3. Category: Quantitative non-randomized | Quality assessment | ||||||||
| Authora, year | Are there clear research questions? | Do the collected data allow to address the research questions? | Are the participants representative of the target population? | Are measurements appropriate regarding both the outcome/ intervention (or exposure)? | Are there complete outcome data? | Are the confounders accounted for in the design and analysis? | During the study period, is the intervention administered (or exposure occurred) as intended? | Total MMAT Score | Percent agreement of authors |
| Robelski [43] | 1 | 1 | 1 | 0 | 1 | 0 | 0 | 4 | 86% |
| (p. 2) | (p. 6f.) | (p. 6) | (p. 7) | (p. 8) | (p. 10) | unclear | |||
| (p. 6) | information on disturbance variables, current time in HO is not collected | ||||||||
| Wegner [21] | 1 | 1 | 1 | 0 | 0 | 1 | 1 | 5 | 86% |
| (p. 15) | (p. 15, 16) | (p. 15) | (p. 16) measurements checked for plausibility, no reliability information | not reported | (S. 15) | (S. 15) | |||
| 4. Category: Quantitative descriptive | Quality assessment | ||||||||
| Authora, year | Are there clear research questions? | Do the collected data allow to address the research questions? | Is the sampling strategy relevant to address the research question? | Is the sample representative of the target population? | Are the measurements appropriate? | Is the risk of nonresponse bias low? | Is the statistical analysis appropriate to answer the research question? | Total MMAT Score | Percent agreement of authors |
| Spinks [44] | 1 | 1 | 1 | 1 | 0 | 0 | 0 | 4 | 86% |
| (p. 249) | (p. 249) Survey: | (p. 249) | (p. 249) | not reported | not reported | not reported | |||
| Ferreira [41] | 1 | 1 | 1 | 0 | 0 | 0 | 0 | 3 | 100% |
| (p. 120) | (p. 124) | (p. 125) | unclear | not reported | not reported | not reported | |||
| (p. 125) | |||||||||
| no description of the sample | |||||||||
| 5. Category: Mixed methods | Quality assessment | ||||||||
| Authora, year | Are there clear research questions? | Do the collected data allow to address the research questions? | Is there an adequate rationale for using a mixed methods design to address the research question? | Are the different components of the study effectively integrated to answer the research question? | Are the outputs of the integration of qualitative and quantitative components adequately interpreted? | Are divergences and inconsistencies between quantitative and qualitative results adequately addressed? | Do the different components of the study adhere to the quality criteria of each tradition of the methods involved? | Total MMAT Score | Percent agreement of authors |
| Janneck [42] | 1 | 1 | 1 | 1 | 0 | 0 | 1 | 5 | 86% |
| (p. 1052) | (p. 1055ff.) | (p. 1055) | unclear | not reported | (p. 1055) | ||||
| (p. 1055) | (p. 1056) | ||||||||
| Steward [45] | 0 | 1 | 1 | 1 | 1 | 0 | 1 | 5 | 86% |
| unclear (p. 104) not clear enough | (p. 105) | (p. 105) | (p. 105) | (p. 105f.) | not reported | (p. 105) | |||
| Steward (2001) [46] | 1 | 1 | 1 | 1 | 1 | 0 | 0 | 5 | 100% |
| (p. 143) | (p. 143) | (p. 142, 143) | (p. 143) | (p. 144f.) | not reported | not reported | |||
Key: yes = 1, no/unclear = 0, NR = not reported, total = MMAT score in total: 1-2 (low quality), 3–5 (moderate quality) and 6–7 (high quality). aOnly the first author’s name for each study is shown here; full author details can be found in the references.