Abstract
Background
Missing outcome data can pose a serious threat to the validity of randomised clinical trial results. We aimed to study the extent of missing outcome data in randomised clinical trials of psychological interventions.
Methods
We performed a retrospective study of randomised clinical trial reports of psychological interventions published in World Psychiatry, JAMA Psychiatry, Lancet Psychiatry, American Journal of Psychiatry, British Journal of Psychiatry, or Psychotherapy and Psychosomatics from 2017 to 2022. We assessed the proportion of missing outcome data, whether missing data patterns differed between types of outcomes, participants, intervention lengths, and psychological intervention types, how missing outcome data were handled in the statistical analyses, and whether trialists discussed missing outcome data in the discussion section of the manuscript.
Results
We identified 182 randomised clinical trials (233 primary outcomes), of which 206 outcomes (88.4%) were assessed at high risk of bias due to missing data. The overall mean percentage of missing outcome data was 18.3% (95% confidence interval (CI): 16.7–20%) for all outcomes. The percentages of missing data were 18.9% (95% CI: 17.1–20.6%; 180 outcomes) for symptom severity scales and 1.8% (95% CI: 2.3–3.3%; 6 outcomes) for ‘hard’ binary outcomes. Trials including participants with borderline personality disorder had the highest percentage of missing outcome data (33.1%; 95% CI: 22.3–43.9%) compared with other psychiatric disorders. Fisher’s exact test showed that intervention lengths and psychological intervention types were associated with the proportion of missing outcome data (p < 0.001), but there were no clear patterns.
Conclusion
Missing outcome data is a considerable problem in randomised clinical trials of psychological interventions, and trialists should consider the corresponding methodological limitations in the design and analysis to reduce the risk of bias due to missing outcome data.
Clinical trial registration number
Not applicable.
Supplementary Information
The online version contains supplementary material available at 10.1186/s12888-024-06263-4.
Keywords: Missing outcome data, Randomised clinical trials, Psychological interventions
Introduction
Although the randomised clinical trial is the gold standard in clinical intervention research, the validity of the trial results may be compromised by missing outcome data. The higher the proportions of missing outcome data in a trial, the more problematic it is to handle statistically, and the more biased the conclusions potentially become [1]. The extent of this potential bias due to missing data depends on the mechanisms causing the data to be missing and the statistical methods applied [2, 3]. Three mechanisms are causing missing data: missing completely at random (MCAR), missing at random (MAR), and missing not at random (MNAR) [2, 4, 5]. Data are MCAR when the mechanism causing the missing data depends neither on the observed data nor the missing data [2, 5], for example if paper questionnaires are lost in the mail or blood samples are accidentally damaged in the lab. MCAR causes inflated standard errors due to the reduced sample size but does not cause bias (‘systematic error’), as the incomplete dataset is representative of the entire dataset [2]. If the missingness mechanism only depends on the observed data, then the missing data are MAR [2]. MAR allows the prediction of the missing values based on the participants with complete data [2]. If the missingness depends on the missing data, and this dependency remains even given the observed data, then data are classified as MNAR [2, 5]. The MAR and MNAR conditions cannot be distinguished based on the observed data because, by definition, the missing data are unknown and it can, therefore, not be confirmed or rejected if the observed data can predict the unknown data [2, 5].
Trial data should be analysed including data from all randomised participants, i.e., the intention-to-treat principle. Omitting participants who do not have complete data from the analysis is known as complete case (or available case) analysis [1]. When only a few observations are missing, little harm will be done. Still, when many are missing it will lead to a loss of statistical power and bias if the characteristics of the participants with missing data are different from the characteristics of the participants with available data. One alternative approach to complete case analysis is to use single- or multiple imputations, whereby missing values are replaced by some plausible estimated values predicted from the available data [1]. The validity of single imputation depends on specific assumptions that the missing values, for example, are identical to the last observed value [5]. These assumptions are often unrealistic and single imputation is, therefore, a potentially biased method and should be used with caution [5–7]. Using multiple imputation instead, multiple plausible imputed data sets are created, and results obtained from each are appropriately combined [2, 3]. Multiple imputation is a valid general method for handling missing data in randomised clinical trials, and this method is available for most types of data [2, 6–8]. However, multiple imputation only leads to valid results if data are not MNAR, and as mentioned, it can never be determined if data are MAR or MNAR. Other valid statistical methods exist for handling missing data in trials [3], but no completely satisfactory solution for handling data MNAR is available, and missing outcome data are always a threat to the validity of trial results.
The extent of missing data could vary based on the nature of outcomes; for instance, conducting interviews or completing questionnaires is time-consuming and resource-intensive, potentially leading to higher rates of uncollected data. Outcomes that are simpler to evaluate, for example all-cause mortality, which can be retrieved through medical records, may lead to lower levels of missing data [9]. Additionally, specific participant demographics, for example impulsive patients or patients with low levels of functioning, may exhibit higher rates of non-attendance at follow-up assessment [10]. The length of trial interventions could also contribute to increased missing data at follow-up – it might be easier for participants to show up for outcome assessment if the trial period is short. The type of psychological intervention may also contribute to the risk of missing data, for example if participants are dissatisfied with the treatment and prematurely drops out of the whole trial [11].
The present study sought to determine the proportion of missing outcome data in randomised clinical trials of psychological interventions published in six high impact factor psychiatry journals; if missing data patterns differed per types of outcomes, participants, intervention lengths; and psychological intervention types; how missing outcome data were handled in the statistical analyses; and if trialists discussed missing outcome data in the discussion section of the manuscript.
Methods
Search strategy and selection criteria
Two investigators (SJ, PF) independently searched for randomised clinical trials of psychological interventions (for example psychodynamic therapy, cognitive behavioural therapy, or mentalisation-based therapy) for any type of mental health disorder published from 2017 to 2022 (all years inclusive) on the websites of six high-journal impact factor psychiatric journals:
World Psychiatry.
Lancet Psychiatry.
JAMA Psychiatry.
Psychotherapy and Psychosomatics.
American Journal of Psychiatry.
British Journal of Psychiatry.
We chose these peer-reviewed journals due to their high impact factor (ranging from 10.5 to 73.3 in Web of Science for the year 2022). The period was chosen to provide an overview of the current research practice. Two investigators (SJ, PF) independently screened titles and abstracts. Any discrepancies were resolved through discussion or, if required, through discussion with a third investigator (JCJ). Full-texts were retrieved for all trial reports.
Eligibility criteria
We included any randomised clinical trial (as defined by trialists) assessing the effects of a psychological intervention for any mental health disorder, i.e. we included (1) trials comparing psychological interventions with non-psychological interventions (for example drugs, no intervention, or wait-list) and (2) trials comparing two or more psychological interventions with each other. We excluded prevention trials assessing healthy participants.
Data extraction
Six independent investigators (SJ, PF, RKA, CBK, JJP, FS) extracted data and performed risk of bias assessments in pairs. The final data extractions and risk of bias assessments were reached through consensus. Any disagreements following the independent data extractions or risk of bias assessments were resolved through discussion or, if required, through discussion with a third investigator (JCJ). Both the published trial reports and supplementary materials were used for data extraction.
We extracted data from the assessment timepoint defined as primary by the trialists. If no primary timepoint was defined by the trialists, we used the timepoint closest to the end of treatment in both groups. We included feasibility studies but only used the clinical outcome(s).
Assessment of risk of bias due to missing outcome data
For each trial, we assessed domain 3 of the Cochrane Risk of Bias tool – version 2 (RoB2) [12], which is “Bias due to missing outcome data”. Low risk of bias was assessed, if outcome data was available for all, or nearly all, participants (usually 95% complete data). High risk of bias was assessed, if missing data exceeded 5%, there was no evidence that the result was not biased, and the missingness could, and was likely to, depend on the true value [12].
Classification of the reporting of the missing data
We classified how the trial reported the extent of the missing data into three categories:
‘Fully reported’: when the trial reported in detail the extent of missing data for the primary outcome.
‘Partially reported’: when the overall proportion of missing data was reported in a flowchart, but the extent of missing data for the primary outcome was unclear.
‘Not reported’: when the extent of missing data was not reported.
Classification of outcomes
We only extracted data for each trial’s primary outcome(s). We classified the different types of primary outcomes as:
‘Hard binary outcomes’: defined as patient-important binary outcomes that are conclusive regarding the disease progression, demonstrating a patient’s feelings, functionality, or survival [13]. Examples of hard binary outcomes are all-cause mortality, suicides, suicide attempts, psychiatric hospitalizations, and self-harm.
‘Symptom severity scales’: defined as any scale (for example interviews or questionnaires) measuring psychiatric symptoms. If a trial assessed a dichotomised version of a continuous outcome (for example response or remission), we also classified this as a symptom severity scale. Examples of symptom severity scales are Hamilton Rating Scale for Depression and Clinical Global Impressions Scale.
‘Count data’: defined as any outcome using countable quantities. Examples of count data are number of binge eating episodes and number of days without drinking alcohol.
‘Other types of outcomes’: any type of outcome not included in the above-mentioned classifications. Examples of other types of outcomes are urine toxicology and other lab results.
Classification of participants
We classified if the trial participants belonged to the following mental health disorder categories: Addiction disorders and comorbidities (for example, substance use disorder and post-traumatic stress disorder; or alcohol use disorder and depression); affective disorders and anxiety disorders; eating disorders and comorbidities; neurodevelopmental disorders; personality disorders; post-traumatic stress disorder; psychosis and schizophrenia spectrum disorders; sleeping disorders; transdiagnostic; or other disorders.
Classification of intervention length
We classified if the intervention lengths were under 1 month; 1–3 months; 3–6 months; 6–12 months; over 12 months; or if there was no information regarding intervention length.
Classification of psychological intervention types
We classified if the psychological trial interventions (both the experimental and control groups) belonged to the following categories: cognitive and behavioural therapies; humanistic therapy; psychodynamic therapy; supportive psychotherapy; systemic/family therapy.
Classification of statistical analyses
To determine the group of participants included in the analyses, we categorised the included population as:
‘Intention-to-treat population’: if all randomised participants with available data were included in the primary analysis;
‘Modified intention-to-treat population’: if participants who did not initiate the interventions were excluded from the analysis;
‘Per protocol population’: If only participants who completed the intervention and follow-up assessments were analysed;
‘Wrong classification of the population‘: if the outcome was reported as being analysed according to the intention-to-treat principle, but the trialists erroneously excluded certain participants from the follow-up assessments and analyses (for example due to protocol violations, adverse effects, or no effects of the intervention). This classification was based on consensus ratings by two review authors (SJ, JJP), as the trialists’ reporting of their analyses was often unclear or ambiguous.;
‘Unclear population’: if it was unclear or not reported which population was included in the primary analysis.
We assessed if the potential impact of missing data was assessed in sensitivity analyses to evaluate the robustness of the primary analyses in trials, where missing data exceeded 5%. Additionally, we grouped the primary statistical analyses used to handle missing data in the following categories: (1) complete case analysis, (2) multiple imputation or similar methods, (3) other types of imputation (for example last observation carried forward, single imputation, worst-case imputation), (4) full information maximum likelihood, (5) regression analyses or similar methods (e.g. mixed-effects modelling, multilevel linear regression, linear mixed-effects models), or (6) unclear/not reported.
Additionally, we assessed whether the trialists discussed the potential strengths or limitations of missing outcome data in the discussion section of the published trial report.
Data analysis
We calculated the mean missing data along with 95% confidence intervals (CI). Confidence intervals of proportions were calculated using one sample proportions test with continuity correction and presented as percentages and 95% CI. We assessed the difference between missing data proportions with Fisher’s exact test. We used the unadjusted threshold (0.05) as the threshold for statistical significance. The analyses were carried out using R version 4.2.1 (R Core Team, Vienna, Austria).
Results
Characteristics of the included trials
A total of 182 randomised clinical trials of psychological interventions were identified, assessing 233 primary outcomes. Table 1 provides an overview of the included trials. The included trials may be found in Supplementary material 1.
Table 1.
Trial characteristics
| Overall n = 182 |
|
|---|---|
| Journal | |
| American Journal of Psychiatry | 18 (9.9%) |
| British Journal of Psychiatry | 27 (14.8%) |
| JAMA Psychiatry | 51 (28.0%) |
| Lancet Psychiatry | 38 (20.9%) |
| Psychotherapy and Psychosomatics | 47 (25.8%) |
| World Psychiatry | 1 (0.5%) |
| Design | |
| Cluster RCT | 3 (1.6%) |
| RCT | 179 (98.4%) |
| Number randomized | |
| median [IQR] | 197.50 [120.00;297.75] |
| mean (95%CI) | 288.81 (230.94;346.68) |
| Type of participants | |
| Addiction disorders and comorbidities | 14 (7.7%) |
| Affective disorders and anxiety disorders | 77 (42.3%) |
| Eating disorders and comorbidities | 9 (4.9%) |
| Neurodevelopmental disorders | 6 (3.3%) |
| Personality disorders | 5 (2.7%) |
| Post traumatic stress disorder | 16 (8.8%) |
| Psychosis and schizophrenia spectrum disorders | 15 (8.2%) |
| Sleeping disorders | 14 (7.7%) |
| Transdiagnostic | 20 (11.0%) |
| Other | 6 (3.3%) |
| Trial intervention length | |
| <1 month | 9 (4.9%) |
| >1–3 month | 74 (40.7%) |
| >3–6 month | 50 (27.5%) |
| >6–12 month | 29 (15.9%) |
| >12 month | 10 (5.5%) |
| No information | 10 (5.5%) |
| Method of collecting primay outcomes | |
| Interview | 63 (34.6%) |
| Questionnaire | 104 (57.1%) |
| Registry | 11 (6.0%) |
| Other | 4 (2.2%) |
| Number of timepoints | |
| 2 | 37 (20.3%) |
| 3 | 74 (40.7%) |
| 4 | 35 (19.2%) |
| 5 | 19 (10.4%) |
| 6 | 7 (3.8%) |
| 8 | 3 (1.6%) |
| 9 | 2 (1.1%) |
| 10 | 2 (1.1%) |
| 11 | 1 (0.5%) |
| 16 | 1 (0.5%) |
| 17 | 1 (0.5%) |
| Number of primary outcomes | |
| 1 | 145 (79.7%) |
| 2 | 25 (13.7%) |
| 3 | 10 (5.5%) |
| 4 | 2 (1.1%) |
| Discussion about missing outcome data in trial report (Yes/No) | |
| No | 92 (50.5%) |
| Yes | 90 (49.5%) |
Risk of missing outcome data bias
Of the 233 outcomes, 206 outcomes (88.4%) were assessed at high risk of bias due to missing data. 27 outcomes (11.6%) were assessed as at low risk of bias (Table 2).
Table 2.
Outcome characteristics
| Overall n = 233 |
|
|---|---|
| Type of primary outcome | |
| Count data | 29 (12.4%) |
| Hard binary | 6 (2.6%) |
| Scale | 196 (84.1%) |
| Other | 2 (0.9%) |
| Table 2. Statistical analyses in the included trials | |
| Intention-to-treat | 58 (24.9%) |
| Modified intention-to-treat | 12 (5.2%) |
| Per protocol | 16 (6.9%) |
| Wrong classification | 129 (55.4%) |
| Unclear/Not reported | 18 (7.7%) |
| RoB2 missing data item | |
| High | 206 (88.4%) |
| Low | 27 (11.6%) |
| Missingness | |
| Fully reported | 93 (39.9%) |
| Not reported | 16 (6.9%) |
| Partially reported | 124 (53.2%) |
| Proportion of missing data | |
| 0–1% | 8 (3.4%) |
| 1–5% | 21 (9.0%) |
| 5–10% | 28 (12.0%) |
| 10–20% | 73 (31.3%) |
| 20–30% | 46 (19.7%) |
| >30% | 38 (16.3%) |
| No information | 19 (8.2%) |
| Reasons for missing data provided? | |
| No | 139 (59.7%) |
| Yes | 94 (40.3%) |
Proportion of missing outcome data in randomised clinical trials of psychological interventions
The overall mean percentage of missing data was 18.3% (95% confidence interval: 16.7–20%) for all primary outcomes (Fig. 1). The percentage of missing data was fully reported for 93 outcomes (39.9%), partially reported for 124 outcomes (53.2%), and not reported for 16 outcomes (6.9%) (Table 2). Figure 1 includes data for the fully reported and partially reported outcomes.
Fig. 1.
Total proportion of missing outcome data in randomised clinical trials of psychological interventions
Missing outcome data stratified by type of outcome
Fisher’s exact test showed that the type of outcome was associated with the proportion of missing outcome data (p < 0.001). The percentage of missing data was 18.9% (95% CI: 17.1–20.6%; 180 outcomes) for symptom severity scales; 1.8% (95% CI: 2.3–3.3%; 6 outcomes) for ‘hard’ binary outcomes (all-cause mortality, self-harm, psychiatric hospitalizations, and out-of-home placement) [14–19]; and 17.8% (95% CI 12.7–22.9%; 27 outcomes) for count data outcomes (Fig. 2a).
Fig. 2.
Missingness stratified by (a) type of outcome; (b) type of participants (c) intervention length, (d) type of psychological intervention
Missing outcome data stratified by type of participants
Fisher’s exact test showed that the type of diagnosis was associated with the proportion of missing outcome data (p < 0.001). The percentage of missing outcome data was 33.1% (95% CI: 22.3–43.9%; 3 outcomes) for participants with personality disorders (all included borderline personality disorder); 19.6% (95% CI 13.2–26.0%; 18 outcomes) for addiction disorders and comorbidities; 18.8% (95% CI 16.4–21.1%; 89 outcomes) for affective disorders and anxiety disorders; 18.7% (95% CI 11.8–25.6%; 8 outcomes) for eating disorders and comorbidities; 13.0% (95% CI -4.3–30.3%; 6 outcomes) for neurodevelopmental disorders; 20.0% (95% CI 15.1–24.9%; 18 outcomes) for post-traumatic stress disorders; 14.8% (95% CI 9.9–20.7%; 15 outcomes) for psychosis and schizophrenia spectrum disorders; 25.3% (95% CI 18.5–32.2%; 24 outcomes) for sleeping disorders; and 12.0% (95% CI 7.7–16.4%; 25 outcomes) for the transdiagnostic category (Fig. 2c).
Missing outcome data stratified by intervention length
Fisher’s exact test showed that intervention lengths was associated with the proportion of missing outcome data (p < 0.001). The percentage of missing data was as follows: 14.5% (with a 95% confidence interval [CI] from 3.9 to 25.2%; across 11 outcomes) for interventions shorter than one month; 17.7% (95% CI 15.3–20.2%; covering 98 outcomes) for those lasting one to three months; 20.9% (95% CI 17.4–24.5%; for 50 outcomes) for durations of four to six months; 18.6% (95% CI 14.9–22.4%; across 35 outcomes) for six to twelve months; and 19.5% (95% CI 10.3–28.6%; involving 10 outcomes) for interventions extending beyond twelve months (Fig. 2b).
Missing outcome data stratified by psychological intervention type
Fisher’s exact test showed that the type of psychological intervention was associated with the proportion of missing outcome data (p < 0.001). The percentage of missing data was 35% (95% CI 34–36%; 176 outcomes) for cognitive and behavioral therapies; 26% (95% CI 21–33%; 2 outcomes) for humanistic therapy; 24% (95% CI 21–28%; 14 outcomes) for psychodynamic therapy; 13% (95% CI 11–15%; 11 outcomes) for supportive psychotherapy; 5% (95% CI 4–7%; 7 outcomes) for systemic/family therapy (primarily due to two trials using ‘hard’ binary outcomes with little to no missing data) (Fig. 2d).
Missing outcome data stratified by other interventions (e.g. different types of control interventions) are presented in the Supplementary material 2.
Statistical analyses of the outcome data and statistical procedures used to handle missing outcome data
In 58 outcomes (24.9%) an intention-to-treat population was analysed; in 12 outcomes (5.2%) it was a modified intention-to-treat population; in 16 outcomes (6.9%) it was a per protocol population; in 129 outcomes (55.4%) a wrong classification of the population was used; and in 18 outcomes (7.7%) the analysed population was unclear (Table 2).
Most trials used either regression analyses (or similar methods) (67 outcomes, 28.8%) or multiple imputation (or similar methods) (69 outcomes, 29.6%) to statistically handle missing outcome data in their primary analysis.
The robustness of the findings was explored with a sensitivity analysis for 48 outcomes (20.6%). The most common sensitivity analysis was multiple imputation (29 outcomes, 12.4%) (Table 3). Statistical analysis methods stratified by sensitivity analysis can be found in Supplementary material 3.
Table 3.
Statistical analysis methods
| Overall n = 233 |
|
|---|---|
| Primary analysis - method | |
| Complete case analysis | 16 (6.9%) |
| Full information maximum likelihood | 4 (1.7%) |
| Multiple imputation or similar methods | 69 (29.6%) |
| Other types of imputation (Last observation carried forward) | 11 (4.7%) |
| Other types of imputation (Single imputation) | 6 (2.6%) |
| Other types of imputation (Worst-case imputation) | 4 (1.7%) |
| Regression analyses or similar methods | 67 (28.8%) |
| Unclear/Not reported | 56 (24.0%) |
| Sensitivity analysis | |
| No | 185 (79.4%) |
| Yes | 48 (20.6%) |
|
Overall n = 48 |
|
| Type of sensitivity analysis | |
| Complete case | 2 (4.2%) |
| Last observation carried forward | 5 (10.4%) |
| Multiple imputation | 29 (60.4%) |
| Single imputation | 1 (2.1%) |
| Unclear | 11 (22.9%) |
Discussion of missing outcome data in the published trial report
90 trials (49.5%) discussed the strengths or limitations of missing outcome data in their published trial report. The remaining trials did not discuss missing outcome data in the trial report.
Discussion
Missing outcome data is a considerable problem in randomised clinical trials of psychological interventions published in major psychiatry journals. We identified a total of 182 randomised clinical trials assessing 233 primary outcomes and found a mean proportion of missing outcome data of 18.3% across all outcomes. 206 outcome results (88.4%) were assessed at high risk of bias due to missing data.
Of the 233 primary outcome, 180 outcomes (77%) were symptom severity scales with a mean percentage of missing data of 18.9%. Six outcomes (2.6%) were hard binary outcomes with a mean percentage of missing data of 1.8%. Trials including participants with borderline personality disorder had the highest proportion of missing data (33.1%) compared with other mental health disorders. The intervention lengths seemed to affect the percentage of missing outcome data, but no clear missing data pattern could be identified. The percentage of missing data was highest for cognitive and behavioural therapies (35%) compared with other psychological interventions, but this result must be considered exploratory as the number of outcomes assessing different psychological interventions varied a lot (e.g. cognitive and behavioural therapies were assessed in 176 outcomes and humanistic therapy was assessed in two outcomes).
Regression analyses or similar methods (67 outcomes, 28.8%) and multiple imputation or similar methods (69 outcomes, 29.6%) were frequently used to statistically handle missing data in the primary analysis. The robustness of trial results was only explored with a sensitivity analysis for 48 outcomes (20.6%). 90 trials (49.5%) discussed the potential bias risk from missing data in the published trial report.
The results from the present study may inform future outcome selection and measurement in psychological trials to avoid bias arising from missing outcome data. Considering the course rule of thumb that missing data exceeding 5% is considered problematic [3, 12], the total proportion of missing data of 18.3% found in this study must warrant careful concern. Based on our results, it is advisable to incorporate ‘hard’ binary outcomes into the outcome hierarchy of a trial to minimize missing outcome data. Examples of ‘hard’ binary outcomes relevant for mental health populations could be all-cause mortality, serious adverse events, suicides, suicide attempts, serious self-harm incidents, employment status, on social welfare or not, or out-of-home placement (for children). These outcomes can be retrieved through medical records or public registries by blinded research personnel, thus minimizing the risk of missing information. Using ‘hard’ binary outcomes as primary outcomes would often increase the sample size. Still, sample sizes based on hard outcomes in psychological trials are achievable, as demonstrated in other areas of medicine [20]. Rather than conducting numerous small trials with flawed methodology, it would be more sensible to focus on fewer trials with larger sample sizes and sound methodology [20].
Our results indicate that patients with personality disorders (borderline personality disorder in particular) had the highest missing data proportion compared with other mental health disorders. This result is similar to a previous systematic review assessing dropout rates from psychotherapy trials for borderline personality disorder, where up to 28.2% dropout rates were found for outpatient settings [21]. We speculate that many patients who drop out of the intervention also do not attend follow-up assessments. The typical reasons for dropout, including dissatisfaction with treatment, expulsion from treatment, and lack of motivation, may also explain the high missing data proportions found in this study. Recent recommendations for randomised clinical trials in personality disorders recognized missing outcome data as a serious limitation in the field [22, 23].
The present study has limitations. First, if a trial assessed outcomes at various timepoints, we evaluated the proportion of missing data from the trialists designated primary timepoint. If the trialists did not specify a primary timepoint, we evaluated data from the timepoint closest to ‘end of treatment’. This methodology was chosen based on the assumption that the trialists’ primary focus would align with this timepoint. However, it can be argued whether ‘end of treatment’ represents a clinically meaningful timepoint for patients, and that the assessment of beneficial and harmful intervention effects possibly are more patient important at longer-term follow-up. Second, the missingness in trials reporting more than one primary outcome may be correlated, which we did not consider in our analyses. Third, we did not publish a protocol and a statistical analysis plan prior to this retrospective study of published trial reports, as we judged that publishing a protocol would not alter the final results of this review. Fourth, we may have missed important trials with different missing data patterns due to our selection of journals. Journals with high journal impact factors are known to have lower risks of bias compared to journals with low journal impact factor [24, 25]. Hence, there is a possibility that the trials we included may have underestimated the ‘true’ bias risk arising from missing outcome data, and that the proportions of missing data are in fact even higher. Fifth, we only focused on psychological interventions although trials assessing other interventions (for example pharmacological interventions) for mental health patients may have comparable problems with missing outcome data [26, 27]. Sixth, we explored missing data patterns in types of outcomes, participants, intervention lengths, and psychological intervention types resulting in multiple comparisons. Still, we did not adjust the thresholds for significance according to multiple testing. Hence, the results of our stratified analyses must be considered hypothesis-generating only.
In sum, some recommendations for optimizing data acquisition in psychological trials can be made based on the results of this study. Trialists should consider incorporating hard binary outcomes into their outcome hierarchy, particularly in trials assessing personality disorders at high risk of missing outcome data. Furthermore, we recommend trialists to optimise their use and reporting of sensitivity analyses, for example by exploring the impact of missing outcome data on the results by testing the missing data mechanisms, i.e. if data seem to be missing under MCAR, MAR, or MNAR. In the present review, the robustness of the findings was only reported as being explored with a sensitivity analysis for 48 outcomes (20.6%) – a result, which is similar to somatic trials [28]. As mentioned, it is impossible to differentiate the MAR and MNAR assumptions. However, a simple way to differentiate the MCAR from MAR and MNAR assumptions could be to create two additional baseline-tables: a baseline table that only includes participants with available outcome data, and another that only includes participants with missing outcome data. By visually inspecting these two baseline tables, it may be possible to judge whether the two tables seem similar, i.e., if data are MCAR or MAR/MNAR.
Conclusion
Missing outcome data is a considerable problem in randomised clinical trials of psychological interventions. Missing outcome data was lowest for hard binary outcomes and highest for symptom severity scales. Participants with borderline personality disorder had the highest proportion of missing data. These results should be considered when designing trials of psychological interventions. Trialists should improve the use and reporting of sensitivity analyses to explore the robustness of the primary analyses for outcomes at high risk of missing outcome data bias.
Electronic supplementary material
Below is the link to the electronic supplementary material.
Acknowledgements
Not applicable.
Author contributions
SJ and JCJ initiated the study and wrote the first manuscript draft. SJ and PF screened the journal websites for eligible trials. SJ, PF, RKA, JJP, CBK and FS independently extracted data and performed risk of bias assessments. MHO performed statistical analyses. All authors read, commented on, and approved the final manuscript.
Funding
This study received no specific funding. Some authors receive salaries from the Copenhagen Trial Unit, Centre for Clinical Intervention Research which is funded by the Danish state.
Data availability
No datasets were generated or analysed during the current study.
Declarations
Ethics approval and consent to participate
The analyses in this study were performed on publicly available, published trial data that have already been approved by the respective ethical committees that originally assessed the included trials. Hence, it was not applicable to obtain ethical approval nor participation consent, e.g. in accordance with the Declaration of Helsinki, for this study.
Consent for publication
Not applicable.
Competing interests
The authors declare no competing interests.
Footnotes
Publisher’s note
Springer Nature remains neutral with regard to jurisdictional claims in published maps and institutional affiliations.
References
- 1.Altman DG, Bland JM. Missing data. BMJ. 2007;334(7590):424. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 2.Sterne JA, White IR, Carlin JB, Spratt M, Royston P, Kenward MG et al. Multiple imputation for missing data in epidemiological and clinical research: potential and pitfalls. BMJ. 2009;338. [DOI] [PMC free article] [PubMed]
- 3.Jakobsen JC, Gluud C, Wetterslev J, Winkel P. When and how should multiple imputation be used for handling missing data in randomised clinical trials–a practical guide with flowcharts. BMC Med Res Methodol. 2017;17(1):162. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Little RJ, D’Agostino R, Cohen ML, Dickersin K, Emerson SS, Farrar JT, et al. The prevention and treatment of missing data in clinical trials. N Engl J Med. 2012;367(14):1355–60. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 5.Dziura JD, Post LA, Zhao Q, Fu Z, Peduzzi P. Strategies for dealing with missing data in clinical trials: from design to analysis. Yale J Biology Med. 2013;86(3):343. [PMC free article] [PubMed] [Google Scholar]
- 6.Zhang Y, Alyass A, Vanniyasingam T, Sadeghirad B, Flórez ID, Pichika SC, et al. A systematic survey of the methods literature on the reporting quality and optimal methods of handling participants with missing outcome data for continuous outcomes in randomized controlled trials. J Clin Epidemiol. 2017;88:67–80. [DOI] [PubMed] [Google Scholar]
- 7.Jørgensen AW, Lundstrøm LH, Wetterslev J, Astrup A, Gøtzsche PC. Comparison of results from different imputation techniques for missing data from an anti-obesity drug trial. PLoS ONE. 2014;9(11):e111964. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8.Horton NJ, Lipsitz SR. Multiple imputation in practice: comparison of software packages for regression models with missing variables. Am Stat. 2001;55(3):244–54. [Google Scholar]
- 9.Mendelsohn AB, Dreyer NA, Mattox PW, Su Z, Swenson A, Li R, et al. Characterization of missing data in clinical registry studies. Ther Innov Reg Sci. 2014;49(1):146–54. [DOI] [PubMed] [Google Scholar]
- 10.Fenger M, Mortensen EL, Poulsen S, Lau M. No-shows, drop-outs and completers in psychotherapeutic treatment: demographic and clinical predictors in a large sample of non-psychotic patients. Nord J Psychiatry. 2011;65(3):183–91. [DOI] [PubMed] [Google Scholar]
- 11.Fernandez E, Salem D, Swift JK, Ramtahal N. Meta-analysis of dropout from cognitive behavioral therapy: magnitude, timing, and moderators. J Consult Clin Psychol. 2015;83(6):1108–22. [DOI] [PubMed] [Google Scholar]
- 12.Sterne J, Savović J, Page MJ, Elbers RG, Blencowe NS, Boutron I, et al. RoB 2: a revised tool for assessing risk of bias in randomised trials. BMJ. 2019;366:l4898. [DOI] [PubMed] [Google Scholar]
- 13.Grey P, Grey A, Bolland MJ. Outcomes, interventions and funding in randomised research published in high-impact journals. Trials. 2018;19(1):592. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14.Cheryl A, King P, corresponding author1 Alejandra Arango, MS,1 Anne Kramer, LMSW,1 Danielle Busby, PhD,1 Ewa Czyz, PhD,1 Cynthia Ewell Foster, PhD,1 and, Gillespie BW. PhD2, for the YST Study Team. Association of the Youth-Nominated Support Team Intervention for Suicidal Adolescents With 11- to 14-Year Mortality Outcomes. JAMA Psychiatry. 2019;76(5):492–8. [DOI] [PMC free article] [PubMed]
- 15.Tinland A, Loubiere S, Mougeot F, Jouet E, Pontier M, Baumstarck K, et al. Effect of Psychiatric Advance directives facilitated by peer workers on compulsory admission among people with Mental illness: a Randomized Clinical Trial. JAMA Psychiatry. 2022;79(8):752–9. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16.O’Connor RC, Ferguson E, Scott F, Smyth R, McDaid D, Park AL, et al. A brief psychological intervention to reduce repetition of self-harm in patients admitted to hospital following a suicide attempt: a randomised controlled trial. Lancet Psychiatry. 2017;4(6):451–60. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17.Fonagy P, Butler S, Cottrell D, Scott S, Pilling S, Eisler I, et al. Multisystemic therapy versus management as usual in the treatment of adolescent antisocial behaviour (START): a pragmatic, randomised controlled, superiority trial. Lancet Psychiatry. 2018;5(2):119–33. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 18.Cottrell DJ, Wright-Hughes A, Collinson M, Boston P, Eisler I, Fortune S, et al. Effectiveness of systemic family therapy versus treatment as usual for young people after self-harm: a pragmatic, phase 3, multicentre, randomised controlled trial. Lancet Psychiatry. 2018;5(3):203–16. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 19.Gillard S, Bremner S, Patel A, Goldsmith L, Marks J, Foster R, et al. Peer support for discharge from inpatient mental health care versus care as usual in England (ENRICH): a parallel, two-group, individually randomised controlled trial. Lancet Psychiatry. 2022;9(2):125–36. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 20.Juul S, Siddiqui F, Faltermeier P, Petersen JJ, Kamp CB, Jakobsen RH et al. Less focus on symptom scales in psychiatric trials: it is time to ensure research equality between psychiatry and other medical specialities. Lancet Reg Health – Eur. 2024;43. [DOI] [PMC free article] [PubMed]
- 21.Iliakis EA, Ilagan GS, Choi-Kain LW. Dropout rates from psychotherapy trials for borderline personality disorder: a meta-analysis. Personal Disord: Theory Res Treat. 2021;12(3):193. [DOI] [PubMed] [Google Scholar]
- 22.Stoffers-Winterling J, Kongerslev M, Simonsen E, Lieb K, Storebø OJ. Clinical trials in personality disorders: recommendations for study design and conduct. Personal Dis: Theor Res Treat. 2023;14(1):127. [DOI] [PubMed] [Google Scholar]
- 23.Juul S. Improving the methodological quality of randomized clinical trials assessing psychotherapy for borderline personality disorder: recommendations for the future. Front Psychiatry. 2022;13:1053844. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 24.Bala MM, Akl EA, Sun X, Bassler D, Mertz D, Mejza F, et al. Randomized trials published in higher vs. lower impact journals differ in design, conduct, and analysis. J Clin Epidemiol. 2013;66(3):286–95. [DOI] [PubMed] [Google Scholar]
- 25.Gluud LL, Sørensen TI, Gøtzsche PC, Gluud C. The journal impact factor as a predictor of trial quality and outcomes: cohort study of hepatobiliary randomized clinical trials. Am J Gastroenterol. 2005;100(11):2431–5. [DOI] [PubMed] [Google Scholar]
- 26.Kamp CB, Petersen JJ, Faltermeier P, Juul S, Siddiqui F, Barbateskovic M et al. Beneficial and harmful effects of tricyclic antidepressants for adults with major depressive disorder: a systematic review with meta-analysis and trial sequential analysis. BMJ Mental Health. 2024;27(1). [DOI] [PMC free article] [PubMed]
- 27.Jakobsen JC, Katakam KK, Schou A, Hellmuth SG, Stallknecht SE, Leth-Møller K, et al. Selective serotonin reuptake inhibitors versus placebo in patients with major depressive disorder. A systematic review with meta-analysis and Trial Sequential Analysis. BMC Psychiatry. 2017;17(1):58. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 28.Wood A, White I, Thompson S. Are missing outcome data adequately handled? A review of published randomized controlled trials in major medical journals. Clin Trials. 2004;1(4):368–76. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Data Availability Statement
No datasets were generated or analysed during the current study.