Disclosure of researcher allegiance in meta-analyses and randomised controlled trials of psychotherapy: a systematic appraisal

Abstract

Objective

Psychotherapy research may suffer from factors such as a researcher’s own therapy allegiance. The aim of this study was to evaluate if researcher allegiance (RA) was reported in meta-analyses and randomised controlled trials (RCTs) of psychotherapeutic treatments.

Design

Systematic approach using meta-analyses of different types of psychotherapies.

Data sources

Medline, PsycINFO and Cochrane Database of Systematic Reviews.

Methods

We evaluated meta-analyses of RCTs regarding various types of psychotherapies. Meta-analyses were eligible if they included at least one RCT with RA and they were published in journals in Medline, PsycINFO and Cochrane Database of Systematic Reviews with an impact factor larger than 5.

Results

We identified 146 eligible meta-analyses that synthesised data from a total of 1198 unique RCTs. Only 25 of the meta-analyses (17.2%) reported allegiance and only 6 (4.1%) used a proper method to control its effect. Of the 1198 eligible primary RCTs, 793 (66.3%) were allegiant. Authors in 25 of these 793 RCTs (3.2%) reported their allegiance while only one study (0.2%) controlled for its effect.

Conclusions

The vast majority among a group of published meta-analyses and RCTs of psychotherapeutic treatments seldom reported and evaluated the allegiance effect. The results of the present study highlight a major lack of this information in meta-analyses and their included studies, though meta-analyses perform slightly better than RCTs. Stringent guidelines should be adopted by journals in order to improve reporting and attenuate possible effects of RA in future research.

Strengths and limitations of this study.

• Researcher allegiance is widely discussed as a potential factor that influences a researcher's actions and the reporting of results in the conducted studies. However, information on the reporting of allegiance in published meta-analyses has not yet been systematically estimated.

• This is the first research article that systematically evaluates the reporting of researcher allegiance in a large scale dataset of 146 meta-analyses and 1198 unique randomised controlled studies of psychotherapy for a broad range of outcomes.

• The criterion of selecting eligible meta-analyses based on a journal's impact factor must be considered with caution.

Introduction

The researcher allegiance (RA) effect is of special concern in studies that are designed to evaluate the treatment effectiveness of different forms of psychotherapy,^1–8 as the investigator may portray allegiances in particular therapies which are correlated with the pattern of the results.^{9 10} RA has been defined as a researcher’s ‘belief in the superiority of a treatment and in the superior validity of the theory of change that is associated with the treatment’ (p55).³ Psychotherapy research was probably one of the very first fields that conceptualised potential allegiance effects for clinical interventions.¹¹ Luborsky et al^{12 13} have shown that RA accounted for two-thirds of the variance in treatment effect in favour of the preferred treatment. Similar potential personal expectations and financial relationships favouring positive results have also been found to affect biomedical research.^{14 15}

The contamination of RA in the psychotherapy era is a long-standing debate. Meta-analyses have found larger effect estimates in psychotherapy studies when RA is observed.^16–18 These effects are attenuated when appropriate statistical methods for controlling for RA are performed.^{1 4 6 12 13 16–22} The aforementioned findings led some researchers to support the existence of allegiance bias,^{1 12 13 19 21 23} which overestimates the effect and threatens the validity of the clinical trials.^{1 21 23} On the other hand, other researchers argue that RA should be viewed as a reflection of the true differences among psychotherapies boosted by the clinical and research expertise,^{24 25} and cannot be considered as existence of bias per se.³ Statistical correction for the presence of allegiance is therefore pointless and may introduce bias.²⁵ This is supported by additional meta-analyses which have shown that RA did not divert the relative treatment effect, concluding that RA was not an important source of bias.^26–28

Allegiance is an essential topic and—bias or not—related researchers seem at least to agree that it should be taken into account effectively. Several sources of allegiance have been provided in order to clarify how allegiance could affect the outcome in randomised controlled trials (RCTs). These could include poor training of therapists, the enthusiasm of the researcher for a particular treatment and the ‘file drawer phenomenon’.³ Furthermore, the nature of psychotherapy, in contradiction to pharmacotherapy, is very difficult to study. Methodological weakness such as wait-list control groups, single group designs, small samples and subjective measurement of clinical improvement may allow RA to interfere.²⁹ Along with the fact that, in the field of psychotherapy, double-blind studies cannot be applied, RA may influence a researcher's actions and its reporting in the conducted studies which could be considered as a potential non-financial conflict of interest.³⁰ However, this type of allegiance bias is not easily detectable. Recently, a new mechanism has suggested that the RA effect may occur partly whenever researchers select biased therapists in study designs.³¹ It is also a fact that RA could affect the outcome whenever researchers select study therapists who share the RA according to the true efficacy hypothesis.¹⁹ Similar mechanisms could occur regarding meta-analyses and studies selection.²⁹ Meta-analysis reflects the potential methodological deficits of the primary studies due to the presence of RA. Thus, meta-analyses could display the same methodological deficits as the primary studies in meta-analysis design, data analysis and interpretation of results because of RA by the authors of the meta-analysis. The developers of some specific psychological treatments may show more interest in the evidence-based practice of their own therapies than in others.³⁰ The RA of authors of a meta-analysis is found to correlate with the outcomes of the meta-analysis.^{6 23}

Although the authors of meta-analyses are required to evaluate all potential biases by the broadly used guidelines,^{32 33} there are no specific guidelines in psychotherapy meta-analyses about clearly addressing the problem of RA. RA is an important factor in showing the benefits of a preferred treatment and therefore attention should be given when interpreting the results of RCTs (eg, therapist allegiance). Moreover, without reporting RA in meta-analyses, the evaluation of the evidence derived may be limited. It is reasonable to assume that neglecting to report RA could be considered as a methodological issue.

In this study we aim to investigate systematically the extent of reporting RA in meta-analyses of RCTs of psychotherapy treatments as well as in the primary RCTs included in these meta-analyses. We searched journals with a relative high impact factor (IF) and enhanced our sample size by including studies from the Cochrane Database of Systematic Reviews (CDSR). Our hypothesis is that RA is not reported in both meta-analyses and their included RCTs. We also hypothesised that the allegiance effect is assessed in only a few meta-analyses and RCTs.

Methods

Selection of meta-analyses

We conducted a comprehensive literature search of meta-analyses in the psychotherapy field published from January 1977 to December 2012. We searched PubMed, PsycINFO and the CDSR using the following search algorithm: (meta-analysis OR systematic review) AND (psychotherapy OR psychoanalysis OR psychological interventions). Both Medical Subject Headings (MeSH) terms and text words were used. From CDSR we selected the most recent version of previously published psychotherapy reviews. The last update was performed in December 2012. To increase the yield of our electronic search, reference lists of all eligible studies and relevant review articles were examined until a comprehensive list was obtained. This systematic review was performed in accordance with the Preferred Reporting Items for Systematic Review and Meta-analyses (PRISMA) statement (on line supplement 1; PRISMA 2009 Checklist).

Eligibility criteria

We considered published meta-analyses of RCTs of psychotherapeutic treatments in journals with a 2012 IF of ≥5 based on Journal Citation Report (JCR, 2012). We selected high impact journals of published meta-analyses to reflect current reporting practices because they have higher reporting standards.^{32 33} The selection criterion of IF has previously been applied to studies of reporting.¹⁵ Meta-analyses of RCTs with at least one study with RA were eligible. We based our decision in order to have a strict criterion since the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) statement requires authors of meta-analyses to report both potential source of bias and conflicts of interest (COIs) of a meta-analysis,³² but does not address the reporting of such biases from included RCTs.

A study was defined as showing RA when one or more of the co-authors had (1) developed the intervention; (2) developed both the therapy and trained the therapists; (3) developed both the intervention and supervised the therapists; (4) supervised and/or trained the therapists alone; or (5) advocated the therapy.² The highest level of allegiance was coded as 5 if the psychotherapy treatment was developed by the author(s) and if they supervised or trained the therapists; 4 if the treatment was developed by the authors but they did not train or supervise the therapists; 3 if the treatment was advocated by one of the authors and they also supervised/trained the therapists; 2 if the treatment was advocated by the authors but they did not train or supervise therapists or in cases where the author showed no advocacy for the psychotherapy but provided better trained or more experienced therapists for one treatment over another; 1 if the treatment was more fully explained in the introduction and/or methods section than the alternative; and 0 if there was no apparent advocacy of one treatment over another. When sufficient information on RA was not given in the full text, we additionally checked the References section in order to identify if a reference was given to previous published research by the same principal investigators showing the efficacy of the treatment relative to no treatment or showing superiority of the treatment compared with other treatment. This procedure has been proposed as a criterion that gives information of moderate to strong allegiance.³¹ We employed these methods because they allow measurement of all direct (eg, developed the therapy) and indirect (eg, author published supporting evidence for the treatment) levels of allegiance in meta-analyses and primary studies.

Meta-analyses should also fulfil the following criteria: (1) evaluate any form of psychotherapy (eg, psychodynamic, interpersonal psychotherapy, cognitive behavioural therapy (CBT), marital therapy, behavioural therapy); (2) assess the direct comparison of (a) different forms of psychotherapy, (b) psychotherapy versus placebo (treatment as usual, waiting list, no treatment), (c) psychotherapy versus medication, (d) group therapy versus individual or group therapy versus group therapy, (e) computer-based psychotherapies versus face-to-face treatments; (3) provide a clear description of the definition of the main outcome or the class of outcomes regarding both mental and medical disorders. Meta-analyses that evaluated a combination of psychotherapies and other alternative treatments (eg, medication) as well as non-bona fide techniques were also included. Meta-analyses with studies pertaining to study designs other than RCTs were excluded.

We excluded meta-analyses which reviewed only the effects of treatments without having a clear verbal component (eg, psychoactive medication, physical exercise) or those concerning non-specific treatments without being compared with a type of psychotherapy (eg, dietary advice, primary care, recreation). We also excluded narrative reviews and systematic reviews without quantitative synthesis of data.

To prevent duplicates of the same studies, if an RCT appeared more than once it was evaluated only in the context of the first publication that described it. We included primary studies only in the English language and in full text format. Two investigators (ED and EE) independently screened both the title and abstract of identified articles as well as full text meta-analyses for eligibility. The decision about the eligibility of a meta-analysis was made independently by the two reviewers and disagreements were reached by consensus.

Data extraction

Eligible meta-analyses, including their primary studies, were reviewed by the first investigator (ED) including disclosure statements, full texts and tables, author's affiliation, acknowledgments, contributors and references, and all online journal supplements. During this review the investigator searched for the presence of RA in each selected meta-analysis, searched for the presence of RA in the RCTs included in each meta-analysis and determined whether or not RA was reported in the meta-analyses and their RCTs. The investigator also established whether any statistical procedure of RA was performed in the meta-analyses and, if so, which method was used according to the definition provided by the authors.

For eligible meta-analyses we extracted the following information: first author, year, journal, IF, supplement issue (Y/N), Cochrane review (Y/N), number of RCTs of psychotherapy treatments included in the meta-analysis, number of articles retrieved and reviewed, RA (Y/N), study population, control/comparison arms, meta-analysis primary outcome reviewed (eg, efficacy, prevention, both efficacy and prevention, effectiveness, harm), type of psychotherapy, meta-analysis authors conducted an allegiant study that was included in the review or advocated the therapy (Y/N) and their number, total number of meta-analysis authors, meta-analysis author allegiance reported (Y/N), meta-analysis author any COI reported (Y/N), meta-analysis author funding sources reported (Y/N), meta-analysis reports on allegiance of included RCTs (Y/N), quality assessment of included RCTs in the meta-analysis (Y/N), and the allegiance assessment method or meta-analytical strategy for allegiance of included RCTs in the meta-analysis (Y/N) (eg, risk of bias, balanced allegiance, subgroup analysis or meta-regression). We employed the actual definition of each meta-analysis in order to categorise the form of psychotherapy treatments (eg, cognitive behavioural therapy, psychoanalytic/psychodynamic informed psychotherapies, family systems therapy). We also used the definition of the meta-analysis author in terms of meta-analysis outcome (eg, effectiveness, efficacy).

From the primary included studies we additionally extracted the following information: first author, year, journal, supplement issue (Y/N), overlapping study (Y/N), RA in primary RCT (Y/N), number of authors with allegiance, total number of included authors in RCTs, RCT authors reported RA (Y/N), RCT authors provided clear data of their allegiances (Y/N), RCT authors reported any COI (Y/N), funding sources of included RCTs reported (Y/N) and, finally, whether the RCT authors used any method of controlling for allegiance according to the definition provided by the authors (eg, balanced allegiance).

One investigator (ED) extracted all the data and another investigator (EE) independently confirmed the extracted data. Disagreements about the above ratings were discussed until a consensus was reached for each meta-analysis. If controversial ratings remained, the data were reassessed by a third party.

Coding of allegiance

Meta-analyses and included RCTs in each eligible meta-analysis were reviewed in order to identify sufficient information to code the potential RA, blind to the results. We used a two-step approach. We first rated evidence of various indicators of RA according to a 6-point scale (from 0 to 5) proposed by Wampold et al,² as mentioned above. We then assigned an absolute allegiance rating of ‘yes’ (for RA=1–5) and ‘no’ (for RA=0) for each psychotherapy treatment. This procedure has the advantage of including all possible indicators of allegiance from strong to weak in both dichotomous and continuous variables.⁴ Any ambiguities about coding of allegiance were solved by discussion until consensus was reached.

Statistical analysis

We report frequencies, medians and IQR. All comparisons were performed using the Fisher exact test and exact test for probabilities. Analysis was performed with Stata 12.0 (STATA Corp, College Station, Texas, USA) and two-sided p values are reported.

Results

Search results

The electronic database search yielded 3347 unique articles of which 146 meta-analyses were eligible (see online supplementary figure S1). Of the 146 meta-analyses, 86 were published in 23 scientific journals (68 specialty psychiatric/psychological journals and 18 general medical journals) and 60 in CDSR (online supplementary references A1–86 and B1–60). The IF of the journals ranged from 5.8 to 14.46 in specialty journals and from 6.4 to 38.8 in general journals. The IF of the Cochrane Database was 5.7.

The 146 selected meta-analyses evaluated a broad spectrum of psychotherapeutic treatments: 25% CBT alone, 50% mixed various psychotherapy treatments in addition to CBT, and 25% all other available forms of psychotherapy alone or in combination. They included 65 studies on treatment effectiveness, 52 on treatment efficacy, 5 on efficacy and effectiveness, 20 on efficacy and prevention and adverse events, 1 on harm and 3 on prevention. Each study included a median of 10 allegiant trials (IQR 7–15) and the median number of participants per meta-analysis was 1202 (IQR 675–2334). Detailed characteristics of selected meta-analyses are presented in online supplementary table S1. The inter-rater agreement was κ=0.94 in meta-analyses and κ=0.91 in RCTs.

Evidence, reporting and assessment of RA in meta-analyses

Evidence of RA in the meta-analysis was present in 56 of 146 (38.4%) eligible meta-analyses. Moreover, 50 of the 56 (89.3%) had at least one author who was a co-author in an included RCT that was allegiant in this specific study (table 1).

Table 1.

Evidence and reporting of allegiance in 146 eligible meta-analyses

Database of published m-a	Total number of m-a	Evidence of allegiance in m-a	Number of m-a with at least one allegiant study conducted by m-a authors	Evidence of at least one allegiant study in included RCTs in m-a	Reporting allegiance of m-a in text	Reporting allegiance in COI section	Reporting allegiance both in main text and in COI section	General reporting of allegiance in m-a	Declaration of interest statement	Reporting of funding sources
PubMed and PsycINFO specialty journals	68	28	26	68	9	0	0	9	34	33
PubMed and PsycINFO General medical journals	18	8	7	18	2	0	0	2	13	15
CDSR	60	20	17	60	2	10	2	14	60	60
Total	146	56	50	146	13	10	2	25	107	108

CDSR, Cochrane Database of Systematic Reviews; COI, conflicts of interest; m-a, meta-analyses; RCT, randomised controlled trial.

In total, only 25 of the 146 meta-analyses (17.2%) discussed or reported RA either in the meta-analysis or in the included RCTs. The level of allegiance was high in 13 meta-analyses and in 12 the level of allegiance was moderate to weak (eg, advocate the treatment). Thirteen of the 146 meta-analyses (8.9%) reported allegiance of the included RCTs in the text. Ten (6.8%) reported some kind of evidence of RA by the author(s) of the meta-analysis in the section on COI, even though all journals required a declaration of competing interests. Finally, two meta-analyses (1.4%) reported allegiance in both the main text and in the section on COI (table 1). The characteristics of the total meta-analyses with COI statements about allegiances are presented in online supplementary table S2. These percentages differ significantly from the 107 of the 146 selected meta-analyses (73.3%) that disclosed potential COI other than allegiance (Fisher exact=92.964, p=1.2×10⁻²⁹) and the 108 of 146 (74.0%) that reported the funding sources (Fisher exact=95.124, p=3.0×10⁻¹⁹). Additionally, only 6 of the 146 meta-analyses (4.1%) evaluated the presence of allegiance. Details per journal as well as the total number of meta-analyses are shown in tables 2 and 3.

Table 2.

Characteristics of selected meta-analyses in specialty psychiatric/psychological journals

Specialty psychiatric/psychological journals (publication date range)	No of published m-a	No of included RCTs	M-a authors included their allegiant studies	M-a authors reporting allegiance of included RCTs	Declaration of interest statement	Reporting of funding sources	Allegiance assessment of included RCTs	Quality assessment of included RCTs
Am J Psychiatry (2011–2005)	6	168	5/6	0/6	5/6	5/6	0/6	4/6
Am Psychol (2006)	1	32	1/1	1/1	1/1	1/1	1/1	0/1
Arch Gen Psychiatry (2004)	2	33	0/2	1/2	0/2	1/2	0/2	0/2
Biol Psychiatry (2010)	1	10	1/1	0/1	1/1	1/1	0/1	0/1
Bipolar Disord (2004–2009)	2	18	2/2	0/2	1/2	1/2	0/2	0/2
Br J Psychiatry (2003–2012)	7	141	4/7	0/7	4/7	5/7	0/7	6/7
Clin Psychol Rev (2001–2012)	22	416	8/22	5/22	4/22	4/22	4/22	12/22
J Am Acad Child Adolesc Psychiatry (2007)	1	11	0/1	1/1	1/1	1/1	0/1	1/1
J Clin Psychiatry (2008–2010)	4	94	1/4	0/4	2/4	2/4	0/4	2/4
Psychol Bull (2006)	1	35	1/1	0/1	0/1	1/1	0/1	0/1
Psychol Med (1994–2011)	16	235	4/16	1/16	12/16	8/16	0/16	14/16
Psychother Psychosom (2007–2012)	5	109	0/5	0/5	3/5	3/5	0/5	5/5
Total (1994–2012)	68	1302	26/68	9/68	34/68	33/68	5/68	44/68

m-a, meta-analyses; RCT, randomised controlled trial.

Table 3.

Characteristics of selected meta-analyses in general medicine journals and the Cochrane Database of Systematic Reviews (CDSR)

General medicine journals and CDSR (publication date range)	No of published m-a	No of included RCTs	M-a authors included their studies	M-a authors reporting allegiance of included RCTs	Declaration of interest statement	Reporting of funding sources	Allegiance assessment of included RCTs	Quality assessment of included RCTs
AIDS (2008)	1	38	1/1	0/1	0/1	1/1	0/1	1/1
Am J Epidemiol (2009)	1	51	0/1	0/1	1/1	1/1	0/1	0/1
Arch Intern Med (1996)	1	23	1/1	0/1	0/1	0/1	0/1	0/1
Arthritis Rheum (2002)	1	25	0/1	0/1	0/1	1/1	0/1	1/1
BMC Med (2009–2010)	2	39	1/2	0/2	2/2	1/2	0/2	2/2
BMJ (1998–2006)	4	57	2/4	0/4	4/4	4/4	0/4	4/4
Eur Heart J (2007–2009)	2	73	0/2	0/2	2/2	2/2	0/2	1/2
Gut (2009)	1	19	1/1	0/1	1/1	1/1	0/1	1/1
Lancet (2004)	1	25	0/1	0/1	1/1	1/1	0/1	1/1
Obes Rev (2011)	1	12	0/1	0/1	1/1	1/1	0/1	1/1
Pain (1999–2010)	3	68	1/3	2/3	1/3	2/3	0/3	3/3
CDSR (2001–2012)	60	995	17/60	14/60	60/60	60/60	1/60	60/60
Total 1996–2012	78	1425	24/78	16/78	73/78	75/78	1/78	75/78
Sum total (1994–2012)	146	2727	50/146 (38.4%)	25/146 (17.2%)	107/146 (73.3%)	108/146 (74.0%)	6/146 (4.1%)	119/146 (81.5%)

m-a, meta-analyses; RCT, randomised controlled trial.

Of the 25 meta-analyses that reported allegiance, 9 were published in specialty journals, 2 were published in general medical journals and 14 were published in CDSR (p=0.26), while the assessment of RA was performed only in specialty journals and in CDSR. Specifically, RA was assessed by different methods (table 4). Finally, 10 of the 25 meta-analyses that reported allegiance were published between 2010 and 2012 while 4 of the 6 studies that used a meta-analytic strategy for its effect were published between 2011 and 2012.

Table 4.

Description of 25 meta-analyses reporting allegiance and 6 meta-analyses reporting estimates of the association between allegiance and outcome

Meta-analysis ID	Author	Year	Journal	Treatment of interest	Age group	Clinical population	Section in text where allegiance of included RCTs reported	Met analyticalstrategy for allegiance	Statistical method	Positive to allegiance bias hypothesis
22466509	Cuijpers et al19	2012	Clin Psychol Rev	NDST	Adults	Depression	Abstract, methods and discussion	Yes	Subgroup analysis (moderator variable)	Yes
21996291	Wampold et al5	2011	Clin Psychol Rev	Mixed ESTs	Any age	Anxiety and depression	Introduction, methods, results, discussion	Yes	Studies quality (moderator variable)	Yes
20547435	Tolin21	2010	Clin Psychol Rev	CBT	Any age	Various mental disorders	Discussion	Yes	Meta-regression (moderator variable)	No
18466666	Smits and Hofmann	2009	Psychol Med	CBT	Adults	Anxiety disorders	Methods and discussion	No	None	None
18055080	Benish et al	2008	Clin Psychol Rev	Mixed	Adults	PTSD	Discussion	No	None	None
18049290	Klein et al	2007	J Am Acad Child Adolesc Psychiatry	CBT	Adolescents	Depression	Methods, results and discussion	No	None	None
16480801	Malouff et al22*	2007	Clin Psychol Rev	PST	Any age	Mental and physical problems	Results and discussion	Yes	Meta-regression (moderator variable)	Yes
17032068	Weisz et al5	2006	Am Psychol	Mixed	Children and adolescents	Various youth problems	Discussion	Yes	ANOVA (moderator variable)	No
15583112	Leichsenring et al	2004	Arch Gen Psychiatry	STTP	Adults	Specific psychiatric disorders	Methods and discusion	No	None	No
12237193	Eccleston et al	2002	Pain	Mixed	Children and adolescents	Chronic pain	Results	No	None	No
10204712	Morley et al	1999	Pain	CBT/BT	Adults	Chronic pain	Results	No	None	None
CD005652	Stoffers et al5	2012	CDSR	Mixed	Adults	Borderline personality disorder	Introduction, methods, results, discussion, tables, COI section	Yes	Risk of bias	Yes
Continued CD007668	Gibbon et al20	2010	CDSR	Mixed	Adults	Antisocial personality disorder	Tables, COI section	No	None	None
CD001027	Price et al34	2008	CDSR	CBT	Adults	Chronic fatigue syndrome	Results	No	None	None
CD004797	Littell et al	2005	CDSR	MST	Adults	Social, emotional, and/or behavioural problems	Tables	No	None	None
CD009514	Smith et al18	2011	CDSR	Relaxation techniques	Adults (only women)	Pain management	COI section	No	None	None
CD004025	Mössler et al13	2011	CDSR	Music therapy	Any age	Schizophrenia	COI section	No	None	None
CD002902	Whalley et al17	2011	CDSR	Mixed	Adults	Coronary heart disease	COI section	No	None	None
CD002014	Henschke et al19	2010	CDSR	CBT/BT	Adults	Chronic low back pain	COI section	No	None	None
CD007944	Roberts et al37	2010	CDSR	Mixed	Any age	Acute traumatic stress symptoms	COI section	No	None	None
CD005378	Ruddy and Dent-Brown	2007	CDSR	Drama therapy	Adults	Schizophrenia	COI section	No	None	None
CD004687	Abbass et al	2006	CDSR	STPP	Adults	Common mental disorders	COI section	No	None	None
CD004717	Crawford-Walker et al	2005	CDSR	Distraction techniques	Any age	Schizophrenia	COI section	No	None	None
CD001134	Dennis and Creedy	2004	CDSR	Mixed	Adults (only women)	Postpartum depression	COI section	No	None	None
CD000524	Jones et al	2004	CDSR	CBT	Any age	Schizophrenia	COI section	No	None	None

*Whether a developer of PST conducted the study under investigation (no term allegiance).

BT, behavioural therapy; CBT, cognitive behavioural therapy; CDSR, Cochrane Database of Systematic Reviews; COI, conflict of interest; EST, empirically supported therapy; MST, multisystemic therapy; NDST, non-directive supportive therapy; PST, problem-solving therapy; PTSD, post traumatic stress disorder; RCT, randomised controlled trial; STTP, short-term psychodynamic psychotherapy.

Evidence, reporting and assessment of RA in included RCTs

The 146 selected meta-analyses synthesised data from a total of 2727 RCTs. Finally, 1198 RCTs were eligible for inclusion in the study. Reasons for exclusion are described in online supplementary figure S2. The median number of authors per RCT was 5 (IQR 3–27). From the 1198 included RCTs, 793 (66.3%) were allegiant studies, 142 (11.8%) were non-allegiant and 263 (21.9%) were defined as unclear, taking into account the absence of clear existence of sufficient information to code RA. In the 793 allegiant RCTs, evidence of author's allegiance through the text was provided in 405 (51.0%) studies while the remaining 388 (49.0%) provided information of allegiance through a cited reference by the same principal investigators to their previous research.

The median number of authors with allegiance was 5 (range 3–7 per RCT). With regard to the indicators of allegiance, in 74 (9.3%) of the 793 included allegiant studies the treatment was developed and the therapists were trained or supervised by the allegiant authors (indicator of allegiance=5), while in 458 (57.8%) the treatment was developed by the allegiant authors (indicator of allegiance=4). In another 85 (10.7%) of the 793 included allegiant studies the treatment was advocated by one of the authors and they also supervised/trained the therapists (indicator of allegiance=3), while in 88 (11.1%) of the 793 allegiant studies the treatment was advocated by the authors but they did not train or supervise the therapists and the author showed no advocacy for the treatment, but provided better trained or more experienced therapists for one treatment over another (indicator of allegiance=2). Finally, in 88 (11.1%) of the 793 allegiant studies the treatment was more fully explained in the introduction and/or methods section than the alternative (indicator of allegiance=1).

Only 25 of the 793 RCTs (3.2%) clearly stated the term ‘allegiance’ and its derivatives, while only one (0.2%) reported allegiance as a potential COI in the relevant section and only one of those (0.2%) assessed the potential effects of allegiance in their analysis. Sixteen of the 25 primary studies that reported allegiance were studies in which the allegiant authors ranked with the highest level of allegiance (eg, developers or developers and trained or supervised the therapists).The comparison between strong allegiant RCTs (16/25) and weak allegiant RCTs (9/25) revealed statistical significant differences in terms of declaration of RA (p=0.003); that is, the stronger allegiant RCTs reported allegiance more often than the weaker allegiant RCTs. In addition, eight of the 25 primary studies that reported allegiance were published between 2005 and 2009 while the rest were published before 2005. Furthermore, 118 (14.9%) of the 793 allegiant RCTs reported a disclosure statement of their potential COIs other than allegiance, while funding sources were reported in 563 (71.2%). The number of primary studies that disclosed allegiance was significantly different from studies that reported any other COI (Fisher’ exact=66.476, p=7.2×10⁻²⁹). On the other hand, 63 of the 405 RCTs that were non-allegiant or of unclear allegiance (15.5%) reported their potential COIs in a disclosure statement while funding sources were reported in 247 of 405 (60.1%) RCTs. Details per journal as well as the total number of included RCTs are described in online supplementary tables S3 and S4.

Discussion

To our knowledge, this is the first systematic appraisal of the reporting of allegiance in RCTs and meta-analyses. We systematically reviewed a large sample size of meta-analyses of RCTs of psychotherapeutic interventions published in high impact scientific journals and their included RCTs. We found that fewer than 20% of meta-analyses reported allegiance and fewer than 10% included a statement about the allegiant author(s) in the meta-analyses in the text or as a disclosure statement elsewhere. Although allegiance was present in two-thirds of the eligible included RCTs, fewer than 5% reported allegiance in the text or as a disclosure statement.

The results of the present study highlight a major gap, given that there is agreement in the literature on the necessity of a more targeted approach on the allegiance effect among researchers and readers.^{1–13 17–31} According to Staines and Cleland,²⁹ “RA represents a major overestimation bias, is a frequent phenomenon applied directly to both primary studies and meta-analyses and additionally meta-analysts, like primary investigators, can exhibit allegiance to a hypothesis being tested”. Poor reporting in meta-analyses and RCTs is crucial because both designs are considered to be the gold standard in evidence-based practice.³⁴ This could imply that many researchers ignore—or, at best, are unaware of—the effect of this phenomenon regardless of the way it influences the results of the studies.^{1–13 17–31 35}

Even in psychotherapy research where allegiance effects have been discussed and conceptualised very early,^{7 11–13 14–18 24 25 28} there is a lack of sensitivity for such potential biases. We found that RA was coded and analysed in a trivial number of meta-analyses. Two plausible explanations can be provided. First, the state of RA is still debatable in terms of possible bias and, although it may be universal in practice settings, its nature effects vary considerably in the literature.^{2 5 6 23–25 31} It is a fact that RA stretches back to the famous Dodo verdict¹¹ and also challenges in terms of a better performance in delivery of treatment.^{1 3 24 25} Very often RA is used as a moderator variable to look at differences between studies.^{2 5 6 19–23 27} Second, information on allegiance is not typically reported by the term ‘allegiance’ in original reports. Moreover, the definition of allegiance differs from study to study. Even if some authors of meta-analyses are familiar with this factor and are willing to investigate RA effects, they have to rely on non-standardised measuring methods³ like reprint analysis (ie, analysis of the publication for the presence of attributes that may hint at allegiance) based on the limited information available in the published articles.^{1 2 4 17 21 31 36} However, it is important to point out that reprint analysis as well as other such procedures including interviews with colleagues of the researchers of interest and interviews with the researchers themselves²⁷ or examinations of previous publications authored by the same work groups³¹ are generally problematic and therefore might lead to erroneous conclusions.³

This study suggests that, without a strict reporting policy, RA is unlikely to be reported in meta-analyses and RCTs. Psychotherapy should move forward, following what is accomplished with pharmaceutical industry trials and sponsorship biases.^{15 32 33 37} Similar statements to the PRISMA statement could be adopted or extended to require authors of meta-analyses to report RA of both meta-analyses and primary studies or report that RA was not disclosed. Meta-analysis authors should report that they have evaluated all potentially relevant sources of bias. Developers of treatments might be encouraged to collaborate with other independent researchers when conducting meta-analyses of their own treatments and any potential personal or financial gain should be disclosed. It is also possible that some researchers may have a financial interest in the treatment—that is, they were involved in training and workshops related to the treatment. However, even if the authors do not have such involvement, they might still benefit from characterisation of their preferred treatment as evidence-based. Other approaches in meta-analyses such as balanced allegiance, subgroup analysis or reporting results by levels of allegiance²⁹ could be adopted as a sensitivity analysis. The investigators of RCTs should have to report their methods (eg, outcome of interest or data analytical methods) before implementation of a clinical trial. Furthermore, the researchers should control for RA by balancing it, at least, when two different psychotherapeutic treatments are compared in a clinical trial. They should also employ another set of researchers to make interpretations of the findings and perhaps, by this method of selecting blind assessors, the RA effects could be minimised. It is important for the psychotherapy field to offer the best reliable guide to policy makers, clinicians and readers, endeavouring to evaluate the relative costs and benefits of choosing a particular therapy over others.

There are some caveats in our study. Our investigation is limited to high impact journals only and therefore we may have not captured meta-analyses and RCTs from other clinically impactful journals on psychotherapy research. However, our collection of journals includes a wide range of journals including several specialised in psychotherapy research such as Clinical Psychology Review, Psychotherapy and Psychosomatics, Psychological Bulletin and Psychological Medicine. Also, this rule applies only to the eligible meta-analyses and not to their included RCTs, ensuring an unbiased sample. Our approach assessed only RCTs included in the eligible meta-analyses and therefore other RCTs in the literature have not been appraised. Nevertheless, there is no evidence that this additional information would be significantly different from our collection of RCTs, and the large number of included studies gives us statistical power to derive solid inferences. A formal comparison of reporting in meta-analyses and RCTs would be of interest. However, the RCTs included here have been published in journals with significantly lower IFs and any possible differences could reflect differences in the reporting practices between high impact and lower impact journals, and not the actual difference between the two designs. Another limitation is the coding of allegiance, since to date there is no unique accepted assessment of RA. However, we tried to measure all possible direct and indirect indicators of allegiance. Other studies in the field have used similar approaches.^{2 17 31 36} Moreover, it has been found that the type of measurement did not alter the strong association between allegiance and treatment effect.²³ Finally, although we found that the majority of allegiant authors of the primary studies had strong allegiance with the preferred treatment (eg, developed it or developed it and supervised/trained the therapist), we have not explored the role of funding sources as an indicator of the existence of allegiance. However, positive effects of treatments were reported by studies without the involvement of the developers or sponsorships.³⁸ Hence, we have only compared the current reporting of allegiance versus the reporting of any other potential factors that influence a researcher's actions and the reporting of results in the conducted studies, such as COIs and funding sources.

In conclusion, we found that the vast majority of meta-analyses and primary RCTs of psychotherapeutic treatments published in high-impact journals failed to report RA. Since allegiance exists in psychotherapy research as an influential factor or bias, revised guidelines might be required for the standardised reporting of this information in a more systematic manner. Indeed, the Cochrane handbook mentions that authors may consider that any potential bias related to the influence of possible bias or potential COIs could be evaluated as an optional ‘other sources of bias’ domain using the risk of bias tool.³⁹ Therefore, coding as well as analysing the effect of RA in every meta-analysis may be the first step in order to clarify the extent of the role of allegiance on psychotherapy outcome research. We believe that a distinct statement of the role of each author's contribution regarding the psychotherapy treatment under investigation similar to the COI statement could be a proper method of standardising the measurement of RA. Potential sources of allegiance should be thoroughly considered, and potential allegiance effects should be extensively discussed by an expert panel in order to consent on specific recommendations on reporting, as in other fields. Optimal clear reporting of any level of RA in RCTs and meta-analyses would improve the transparency of the studies and facilitate the replication of the results.