Abstract
Background:
The presence, types, disclosure rates, and effects of conflicts of interest (COIs) on autism early intervention research have not previously been studied. The purpose of this study was to examine these issues.
Methods:
This study is a secondary analysis of a comprehensive meta-analysis of all group-design, non-pharmacological early intervention autism research conducted between 1970 and 2018. We coded reports for the presence/absence of COI statements, the types of COIs that were disclosed, and for 8 types of COIs, including: (a) the author developed the intervention, (b) the author is affiliated with a clinical provider, (c) the author is employed by a clinical provider, (d) the author is affiliated with an institution that trains others to use the intervention, (e) the author receives payment or royalties related to the intervention, (f) the study was funded by an intervention provider, (g) the study used a commercially available measure developed by the author, and (h) proceeds of the intervention fund the author’s research. Frequencies and proportions were calculated to determine prevalence of COIs and COI disclosures. Meta-analysis was used to estimate summary effects by COI type, and to determine if they were larger than for reports with no coded COIs.
Results:
Seventy percent of reports were coded for ≥ 1 COI, but only ~6% of reports contained COI statements fully accounting for all coded COIs. Meta-regressions did not detect significant influences of any COI type on summary effects, however point estimates for each COI type were larger than for reports with no coded COIs.
Conclusions:
COIs are prevalent but under-reported in autism early intervention research. Improved reporting practices are necessary for researcher transparency, and would enable more robust examination of the effects of COIs on research outcomes.
Keywords: autism, early intervention, conflicts of interest, meta-analysis
Introduction
Conflicts of interest (COIs) occur when researchers stand to benefit from studies that show particular outcomes (Gorman, 2018; Institute of Medicine, 2002). In this circumstance, researcher conduct in service of producing high quality, objective research may be compromised by competing interests such as advancing one’s career or receiving immediate financial incentives (Romain, 2015). Researcher conduct can refer to a variety of activities, such as those involved in designing the study, recruiting and allocating participants to treatment and control groups, data analysis, and interpreting and reporting study findings. Bias stemming from COIs can lead to activities that threaten research integrity, such as preferentially sorting participants into the treatment group, selecting measures that are vulnerable to placebo effects, or engaging in ‘fishing’ or ‘p-hacking’ to search for significant findings (Eisner, 2009; Gandhi et al., 2007; Ioannidis et al., 2014).
COIs can be potential (i.e., benefits have not yet occurred or are not occurring at the time of research but could occur in the future), actual (i.e., benefits have already occurred or are continuing to occur), or perceived (i.e., there are no potential or actual benefits that have accrued or could accrue to the author, but such benefits are a reasonable perception). While the most discussed types of COIs are those that involve direct financial transactions, they also extend to professional or personal relationships, ideological commitments, and religious or political views (Eisner, 2009; Goozner et al., 2009). The potential for biases resulting from the many types of COIs exist regardless of whether researchers believe that their research activities are influenced by COIs (Goozner et al., 2009). The existence of COIs does not necessarily indicate that the competing interests influenced the researcher, only that the potential for bias is present.
In research fields such as medicine, public health, substance abuse prevention, and criminology, meta-analyses have been conducted to investigate the influence of COIs on effect sizes reported in intervention studies. This work has shown that studies conducted by researchers with COIs are more likely to show positive findings than studies conducted by researchers who do not have COIs (Cherla et al., 2019; Chivers, 2019; Delgado & Delgado, 2017; Wells, 2017). In the medical field, where COIs have the longest history of being reported and investigated, COIs are considered ubiquitous and disclosure rates are around 70% of all published studies. Importantly, this number (while high) is probably an underestimate of the true prevalence of COIs in medical research (Cherla et al., 2019; Okike et al., 2009). While other fields have spent time investigating COI prevalence and impacts, very little of this work has been done in autism early intervention research. As a result, there is a dearth of information in this field of study regarding the prevalence of COIs, COI reporting practices, and the extent to which different types of COIs impact findings.
Types of Conflicts of Interest relevant to early intervention autism research
In the context of autism early intervention research, the research outcome most likely to provide researcher benefits resulting from COIs is finding that an intervention has positive effects on children’s development. In the sections below, we discuss several common COIs most relevant to psychosocial (i.e., non-pharmacological) research in this area that could be sources of researcher bias.
The researcher is the intervention developer.
One of the most common COIs in intervention research occurs when the researcher has played a significant role in developing the intervention. In autism intervention research it is standard practice for researchers to lead investigations of their own interventions. This type of COI is associated with affiliation bias (Eisner, 2009). Researchers stand to gain from publishing studies demonstrating the efficacy of their intervention in a variety of ways, including career advancement, increased likelihood of subsequent grant funding, and other benefits that may or may not include direct financial payment. Because of this, intervention developers may be ‘over-invested’ in demonstrating the efficacy of their own designs (Gorman, 2018). Several meta-analyses have found that when intervention developers lead investigations into efficacy of their interventions, findings are indeed systematically biased toward positive effects (Eisner 2009; Gorman 2003; 2005; Gorman & Conde, 2007; Petrosino & Soydan, 2005).
Direct payments.
Researchers may also receive direct financial compensation related to their association with, or expertise in, the intervention they study. This can include payments for materials related to the intervention, such as royalties for manuals or videos offering guidance on implementing the intervention, or for the sale of technologies required to implement the intervention (e.g., computer programs, device applications such as an iPad app, or communication devices). It can also include speaker/consultancy fees for research talks or intervention workshops.
Affiliation with an entity that provides the intervention to children.
In some cases, researchers are employed by or affiliated with an organization, such as a clinic, that provides the intervention to children. These clinics can be private entities, governmental organizations, or attached to a university at which the researcher is employed. The extent to which researchers directly benefit from these affiliations can vary. Direct employment by a clinic likely constitutes a more significant COI than situations involving looser affiliations with university-based clinics, where the researcher’s employment status is not directly tied to the success of the clinic. Still, even in the latter instance, the researcher can gain prestige, career advancement, and future funding opportunities from such affiliations even when they do not involve direct payments to the researcher. In some cases, researchers are associated with organizations that provide the intervention to children, and a portion of the revenue is then filtered back to the researcher to fund future research. This can be through arrangements with private clinical entities, or through university-based centers (Gorman, 2016).
Affiliation with an entity that trains/licenses others to provide the intervention.
Many researchers who study early interventions are employed in university departments that train and license direct support providers who work with young autistic children, such as special educators, board certified behavior analysts (BCBAs), clinical psychologists, speech-language pathologists, and occupational therapists, among others. It is therefore possible that the intervention being studied is part of the curriculum provided to students at the institution. Examples could be clinical psychology programs that train students on the use of developmental practices such as Floortime, or BCBA programs that train students to use Early Intensive Behavioral Intervention. While such associations do not necessarily result in direct payment to the researcher, they can result in indirect payments in the form of salary support or the availability of university funds to support future research made possible by student enrollment (Gorman, 2018).
Commercially available measures developed by the researcher.
Another type of COI is when the researcher is the developer of a standardized, commercially available assessment tool that is used to assess intervention efficacy. Often, the developers of these assessments receive royalties when assessments are sold. When researchers show positive intervention effects via assessment tools that they created, they are able to demonstrate that the assessments are sensitive to change via intervention, which may make them more appealing to other researchers. Therefore, positive intervention effects may indirectly result in financial gain for the researcher.
Disclosure of COIs
In general, it is not an expectation that researchers be free of COIs, but rather that they are transparent in reporting them. Requiring that investigators properly disclose COIs is not a new practice; COI disclosure was first introduced as a requirement for submitting research findings in the early 1980s by the New England Journal of Medicine (Goozner et al., 2009; Relman, 1984). Since then, the Committee on Publication Ethics (COPE) has developed broad definitions of COIs, and guidelines for editors that outline courses of action if it is suspected that researchers have not properly disclosed COIs in their submissions (COPE, 2019). In the last few decades, requiring COI disclosure has become more commonplace, and COPE now has wide membership among psychological journals. However, while COPE provides general guidance, specific definitions and disclosure policies are largely left up to individual journal editors, who may vary widely in their practices. Because there is some ambiguity, and in some cases disagreement, about what constitutes COIs in psychosocial intervention research, COI disclosures appear to be somewhat unregulated in this field as compared to other fields. In fact, others who have examined COIs in some subfields of psychology suspect that the majority of COIs go unreported (Chivers, 2019; Eisner et al., 2015).
In psychology research, failure to disclose COIs appears especially commonplace for activities such as receiving speaker fees for presentations or engaging in paid consultancies. These activities often constitute clear COIs, as they involve direct monetary compensation to researchers. Further, paid speeches are often designed to include descriptions of positive intervention research findings, and the researcher may be invited for a paid speaking/consulting engagement precisely because of positive findings. This has caused some concern, as speaker and consulting fees can be quite high (up to $40,000 in extreme cases) for very prominent researchers (Chivers, 2019).
Reporting that the researcher is the developer of an intervention within a COI statement is also relatively rare. One systematic study of COIs (Eisner, 2015) included 134 reports that examined one of four prominent psychosocial health interventions where an author was also a developer of the intervention. This study found that 92 of these reports (71%) did not contain complete COI statements regarding the researcher’s involvement in intervention development. However, while rare, such disclosure is not unprecedented. Eisner and colleagues (2015) offer an example of a COI statement by Carolyn Webster-Stratton, who is the primary program developer of the Incredible Years, a parenting and social skills program:
“Dr. Webster-Stratton has disseminated these treatments and stands to gain from favorable reports. Because of this, she has voluntarily agreed to distance herself from certain critical research activities, including recruitment, consenting, primary data handling, and data analysis. The University of Washington has approved these arrangements” (Presnall, Webster-Stratton, & Constantino, 2014, cited in Eisner, Humphreys, Wilson, & Gardner, 2015, p. 3).
This statement provides both a description of the relevant COIs, and the measures taken by the research team to reduce the effects of potential bias associated with COIs on research findings.
The current study
To investigate the prevalence and impact of COIs, we used data collected in the context of a comprehensive meta-analysis of early intervention research involving young autistic children. The findings from the main meta-analysis have been published elsewhere (Sandbank et al., 2020). This study examined eight intervention types; behavioral, developmental, naturalistic developmental behavioral interventions (NDBIs), sensory-based, animal-assisted therapy, computer-based, TEACCH, and cognitive behavior therapy. One important conclusion drawn from that study is that a large proportion of autism early intervention research does not meet rigorous quality standards. Researcher COIs could be a potential barrier for improving study quality, making this a worthy topic to explore. In the current study, we use meta-analytic procedures to examine six research questions (RQs) related to COIs in the included studies:
What proportion of studies have at least one of the eight COI types we examined?
What proportion of studies disclosed COIs?
What proportion of COI disclosure statements fully account for coded COIs in a given study?
What types of COIs (among those that are disclosed or undisclosed) were identified in the included studies, and in what proportions (4a)? Were there any associations among COI types (4b)?
Are studies with coded COIs associated with greater summary effect sizes as compared to studies in which a COI cannot be identified?
What are the effects of particular types of COIs on summary effect sizes?
Method
The current study is a secondary analysis of a recently completed comprehensive systematic review and meta-analysis of all early intervention studies for young autistic children (Sandbank et al., 2020). Effect size information was extracted for all outcomes included in each report. The initial search and coding procedures are briefly outlined below; please reference the larger study for a more thorough discussion of our search strategy and coding of intervention and outcome types.
Search
Relevant studies were gathered through a search of nine online databases including Academic Search Complete, Cumulative Index of Nursing and Allied Health Literature (CINAHL) Plus with Full Text, Educational Administration Abstracts, Education Resources Information Center (ERIC), Education Source, MEDLINE, Psychology and Behavioral Sciences Collection, PsycINFO, and SocINDEX with Full Text. The search was conducted with various combinations of the following keywords: (autis* OR ASD OR PDD OR Aspergers) AND (Intervention OR therapy OR teach* OR treat* OR program OR package) AND (Assign* OR “Control group” OR BAU OR “wait list” OR RCT OR Random* OR Quasi OR “treatment group” OR “intervention group” OR “group design” OR trial). In addition to searching these publication databases, the National Database for Autism Research (NDAR), the National Institutes of Health (NIH) Matchmaker, and the Institute of Education Sciences (IES) databases were examined to identify unpublished or grey literature. Finally, a list was compiled of 106 investigators who received federal grants to conduct autism research. Emails were sent to 90 of these investigators with a request to share data that would meet the inclusion criteria. The contact information for the remaining 16 investigators could not be located. No new data was acquired from this process. This process resulted in 12,933 manuscripts that were then subject to abstract screening and full text review.
Screening
A preliminary screening of titles and abstracts was conducted using abstrackr software (Wallace, Small, Brodley, Lau, & Thomas, 2012). After irrelevant manuscripts were discarded, full texts of remaining manuscripts were examined to determine if they met the following inclusion criteria: published in English, publication date between 1970-present, participants were reported to have a diagnosis of autism, the average age of participants ranged from 0–8 years, and studies used group designs that included both a treatment and control group. Refer to Sandbank et al., 2020 for the PRISMA diagram depicting this process. A full list of included reports is available in Appendix S1.
Coding
After screening, a total of 150 reports remained that were then coded for (a) participant, intervention, and outcome characteristics; (b) presence, content, and types of COIs; and (c) effect size information. For the purposes of the current study, we limit our discussion of coding procedures to COIs and effect size information.
Conflicts of interest.
Reports were first examined to determine whether authors disclosed a COI. Reports were coded as either having no COI statement, a statement indicated the authors do not have any COIs (note that it is possible for the authors to have reported ‘no COIs’, but for the manuscript to still be coded as having COIs if we found that COIs did indeed exist), authors reported potential future COI(s), or authors reported COI(s).
Next, studies were coded based on the presence or absence of eight types of COIs. COI categories were constructed by reviewing both the relevant literature and the studies included in the meta-analysis. Categories included: (a) the author developed the intervention (we did not include adaptations of existing interventions, such as when an intervention is adapted for use in a country other than where it was created), (b) the author is affiliated with an entity (usually a university clinic) that provides the intervention as a service to children, (c) the author is an employee of a clinic that provides the intervention as a service to children, (d) the author is affiliated with an institution that trains others to use the intervention, (e) the author receives payment or royalties from the intervention (including materials, speaker fees, paid workshops, etc.), (f) the study was funded by an entity that provides the intervention as a service to children, (g) the study used a measure that was commercially available and developed by a researcher, and (h) proceeds of the intervention fund the author’s research. Definitions and examples can be found in Table 1.
Table 1.
Definitions of Conflict of Interest Categories
COI Category | Definition |
---|---|
Intervention Developer | The researcher developed the intervention, or is part of a team of intervention developers. We included interventions that combined multiple previously developed intervention components in a novel way, but excluded instances where the researcher an existing interventions (e.g., for a population culturally distinct from the population on whom the intervention was developed), and then examined the adaptation. |
Intervention Provider: Institutional Affiliation | The researcher is affiliated with an entity that provides the intervention to children. This usually occurs when the author is employed by a University that also houses a clinic providing the intervention. |
Intervention Provider: Direct Employment | The author is employed by an entity (e.g., a clinic) that offers the intervention as a clinical service. |
Institutional Training | The researcher is employed by an institution (e.g., a University) that trains other to use the intervention. |
Direct Payments | The researcher receives monetary compensation from the intervention, including royalties for the sale of materials (e.g., manuals, software, hardware), speaker fees, and paid workshops. |
Funded by service providers | The study is funded by an entity that provides the intervention to children. |
Intervention Funds Research | Proceeds from the intervention fund the author’s research (e.g., through a university-based center). |
Commercially Available Measure Developed by Author | The study uses a measure that is commercially available, and a researcher is a creator of the measure or has stock in a product used as a measure (e.g., equipment used to measure neural responses). |
Three steps were taken to code COIs. First, if present, COI statements and acknowledgements were examined and coded. Second, methods sections were reviewed to determine if the intervention protocol was developed by the researcher, and to determine if commercially available measures created by the researcher were used to assess intervention outcomes. Third, web searches were conducted on each author associated with the study, the institutions the authors were affiliated with, and the source of funding for the research. Information from these searches were used to identify and code any additional COIs.
The first and second authors independently coded each study on three separate passes for the presence, absence, and type of disclosure statement included in the study text, and the types of COIs relevant to each study. Given the breadth of internet searching required to locate relevant COIs and the number of passes taken for each study, discrepancies between the two coders (which occurred for approximately 40% of the studies after the initial pass) were resolved through a consensus discussion in lieu of computing inter-coder agreement. When agreement could not be reached between the two coders, the full study team was contacted to assist in reaching consensus; this occurred on two occasions. Multiple passes were taken to ensure that the largest number of COIs were located; after the third pass, neither coder located any additional COIs.
Effect size information.
Effect size information extracted for each outcome included post intervention means, standard deviations and sample sizes for the treatment and control groups. This information was used to calculate the standardized mean difference (d) between the two groups, and provided an index that could be comparable across outcome types. The standardized mean differences were then transformed into Hedge’s g to correct for small sample biases (Borenstein, Hedges, Higgins, & Rothstein, 2009).
Analysis
To answer RQs 1–4, we calculated relevant proportions. For RQ 4b, we calculated pairwise Kendall’s tau b (which adjusts for tied ranks) to determine if the presence of particular COIs were associated with the presence of other COIs. To answer RQs 5–6, we first calculated intercept-only summary effects using robust variance estimation (RVE) meta-analysis procedures (Hedges, Tipton, & Johnson, 2010), computed with the robumeta macro in Stata 15 (Statacorp, 2017). This approach accounts for the clustering of effect sizes within studies, which was especially important for the current meta-analysis given the large scope of outcomes included (with some studies reporting more than 100 effect sizes). For RQ 5, we first constructed a dichotomous variable indicating whether any of the COI categories were coded, or if no COI was coded. We then used meta-regression with this variable entered as a predictor to determine if summary effects differed on this variable. For RQ 6, we constructed dichotomous variables for each COI type, indicating if the effect size was coded for a given COI or if no COI was coded. Because we wanted to determine whether each COI type was associated with larger effect sizes as compared to those that were not coded for any COI, effect sizes that were not coded for a given COI type but were coded with other COI types were considered missing. These dichotomous variables were then entered into separate meta-regressions.
Results
The 150 reports included in the meta-analysis comprised 130 unique participant samples (i.e., up to 130 ‘clusters’ were accounted for when deriving summary effects). Across the studies, 1,615 effect sizes were extracted. When calculating proportions for RQs 1–4, we used 150 in the denominator, as these questions had to do with the prevalence of COIs by report (i.e., published manuscript) and not by participant cluster.
Proportions of COIs and COI disclosures
COIs were coded in 105 (70%) of the included reports (RQ1). Of the 150 reports in our study, 102 (68%) did not have a COI statement, 30 (20%) stated that the authors did not have any COIs to disclose, 15 (10%) reported COIs, and three (2%) reported future potential COIs (RQ2). Of the 15 reports that included a COI statement, seven of these statements did not fully account for all of the COIs that we coded. Six COI statements did fully account for all coded COIs, and two reported COIs even though we did not code any COIs. In these cases, COIs were reported that were not discernibly relevant to the study (e.g., a report that disclosed paid consultancies and speaking events with pharmaceutical companies, but the study did not examine pharmaceutical interventions). Thus, of the 105 reports in which a COI was coded, only 5.7% contained COI statements that fully accounted for the coded COIs (RQ3). Of the 18 reports that disclosed an existing or potential future COI, two were behavioral, six were NDBIs, one was developmental, three were computer-based, three were sensory-based, one was TEACCH, and two were classified as ‘other’.
Of the 30 reports that declared no COIs, only seven were not coded for any COIs. The remaining 23 reports each had some identifiable COI. Of the 23 reports that indicated no COIs but were coded as having at least one COI, specific COIs included: (a) researchers developed the intervention (15 reports), (b) researchers were affiliated with an institution that trains others to use the intervention (seven reports), (c) researchers received direct payments or royalties related to the intervention (seven reports), (d) researchers were affiliated with an entity that provides the intervention to children (four reports), researchers were employed by a clinic that provides the intervention to children (three reports), the study was funded by an entity that provides the intervention to children (one report), researchers developed a measure that was used as an outcome measure in the study (one report).
Table 2 shows frequencies and proportions of each of the COI types we coded (RQ4a). Reports in which the researcher was also the intervention developer was the most common COI, followed by affiliation with a clinic that provides the intervention, receipt of direct payments related to the intervention (e.g., royalties, workshop fees), affiliation with an institution that trains others to use the intervention, employment at a clinic that provides the intervention, and the use of a commercially available measure as an outcome variable that was created by the researcher, or for which the researcher owns stock or receives royalties. Studies that were funded by a service provider, and instances where payments for the intervention fund the authors’ research were relatively uncommon. As an additional exploratory analysis, we examined the proportion of reports coded for each COI, broken down by intervention type. This information is shown in Table 3.
Table 2.
Frequency of Each COI Type
COI Category | Number of Reports (%) |
---|---|
Any COI | 105 (70.00) |
Developer | 72 (48.00) |
Clinic Affiliation | 30 (20.00) |
Clinic Employment | 19 (12.67) |
Direct Payments | 29 (19.33) |
Institution Trains Others to Use Intervention | 21 (14.00) |
Researcher Created Measure Used in Study | 8 (5.33) |
Study Funded by Service Provider | 4 (2.67) |
Payments for Intervention Fund Research | 3 (2.00) |
COI = Conflict of Interest
Table 3.
Proportion of Reports with COIs by Intervention and COI Type
COI Type | Intervention Type (Total Number of Reports) |
|||||||
---|---|---|---|---|---|---|---|---|
Animal-assisted Therapy (4) | Behavioral (31) | Cognitive Behavior Therapy (1) | Computer-based Intervention (11) | Developmental (17) | NDBI (38) | Sensory-based Intervention (14) | TEACCH (6) | |
Developer | 0.25 | 0.39 | 0.00 | 0.55 | 0.47 | 0.68 | 0.36 | 0.00 |
Clinic Affiliation | 0.00 | 0.29 | 0.00 | 0.00 | 0.00 | 0.45 | 0.07 | 0.33 |
Clinic Employment | 0.25 | 0.16 | 0.00 | 0.09 | 0.18 | 0.16 | 0.21 | 0.00 |
Direct Payments | 0.25 | 0.03 | 0.00 | 0.00 | 0.47 | 0.29 | 0.21 | 0.00 |
Institution Trains Others to Use Intervention | 0.00 | 0.23 | 0.00 | 0.00 | 0.00 | 0.16 | 0.29 | 0.33 |
Researcher Created Measure | 0.00 | 0.00 | 0.00 | 0.09 | 0.06 | 0.11 | 0.00 | 0.00 |
Study Funded by Service Provider | 0.00 | 0.03 | 0.00 | 0.00 | 0.00 | 0.00 | 0.21 | 0.00 |
Payments for Intervention Fund Research | 0.00 | 0.00 | 0.00 | 0.09 | 0.00 | 0.00 | 0.00 | 0.17 |
Note: COI = Conflict of Interest, NDBI = Naturalistic Developmental Behavioral Intervention. See Sandbank et al., 2020 for descriptions of intervention categories
In terms of associations among COIs, we found several significant correlations (p’s were below < 0.05 for all rτ statistics listed below). Researchers who were intervention developers was correlated with clinic employment (rτ = −0.17), receiving direct payments related to the intervention (rτ = 0.21), and using measures created by an author (rτ = 0.19). Affiliation with an institution that provided the intervention was correlated with clinic employment (rτ = 0.31), and affiliation with an institution that trains others to provide the intervention (rτ = 0.23). Affiliation with an institution that trains others to use the intervention was correlated with receiving direct payments related to the intervention (rτ = 0.19), receiving funding from an intervention provider (rτ = 0.17), and using a measure created by an author (rτ = 0.25). Receiving direct payments was correlated with using a measure created by an author (rτ = 0.18). Finally, proceeds from the intervention funding the authors research was correlated with using measure created by an author (rτ = 0.18).
Meta-regression analyses
The dichotomous variable in the meta-regression comparing summary effects for reports with ‘no coded COI’ to summary effects for ‘any COI’ was not significant (p = 0.22) (RQ5). None of the dichotomous variables in the meta-regressions comparing summary effects for each of the eight COI types to summary effects for ‘no coded COI’ were significant (ps range from 0.13 – 0.57) (RQ6). Summary effects from intercept-only models for each COI category are depicted in a forest plot (Figure 1). Results from meta-regressions can be found in Table 4.
Figure 1.
Forest plot of robust variance estimation (RVE) summary estimates and their respective confidence intervals (CI) by Conflict of Interest (COI) category, where n denotes the number of effect sizes and k denotes the number of studies included in summary effect estimation. Summary effects computed from less than 5 studies should be interpreted with caution.
Table 4.
Results From Robust Variance Estimation Meta-Analyses by COI Type
Intercept-only Model | ||||||
---|---|---|---|---|---|---|
COI Category | n | k | Coef. | SE | p | 95% CI |
Any COI | 1614 | 130 | 0.10 | .08 | 0.22 | [−0.06, 0.26] |
Developer | 1225 | 101 | 0.13 | 0.08 | 0.13 | [−0.04, 0.30] |
Clinic Affiliation | 851 | 70 | 0.11 | 0.10 | 0.27 | [−0.09, 0.31] |
Clinic Employment | 602 | 62 | 0.13 | 0.11 | 0.22 | [−0.09, 0.35] |
Direct Payments | 770 | 69 | 0.06 | 0.10 | 0.54 | [−0.14, 0.27] |
Institution Trains Others to Use Intervention | 701 | 61 | 0.10 | 0.12 | 0.41 | [−0.15, 0.35] |
Researcher Created Measure Used in Study | 524 | 50 | 0.17 | 0.24 | 0.52 | [−0.42, 0.75] |
Study Funded by Provider | 462 | 50 | 0.17 | 0.27 | 0.57 | [−0.61, 0.95] |
Payments for Intervention Fund Research | 463 | 49 | 0.32 | 0.43 | 0.53 | [−1.37, 2.01] |
Note. n = number of effect sizes, k = number of studies; Coef.= Coefficient; COI = Conflict of Interest; CI = Confidence Interval; SE = Standard Error
Discussion
In this study, we conducted a secondary analysis of a previously conducted meta-analysis, which included 150 reports pertaining to autism early intervention research. We examined the prevalence of eight different COI types, and used meta-regressions to determine whether summary effects differed depending on whether these COIs were detected or not. Although none of the variables for COI type were significant, point estimates were larger for effect sizes associated with COIs as compared to effect sizes that were not associated with detectable COIs.
Prevalence of COIs and reporting practices
Our results indicate that, in autism early intervention research, the presence of the COI types we coded is similar to other fields that have extended effort toward accounting for COIs in published literature (Deepa et al., 2019; Okike et al., 2009). Interestingly, several COIs appear to be associated with one another. It is possible that particular COIs enable subsequent COIs; for example, researchers who developed an intervention will have opportunities to publish and collect royalties on intervention manuals. Further, researchers with a given COI (such as being an intervention developer) may be well-connected with, and more likely to collaborate with, researchers with other COIs (such as a researcher who has created a measure that could be used to examine study outcomes).
The reported COIs in this literature are far lower than COI prevalence, as other autism researchers have suggested (Dawson, 2006; 2015). This appears to be true across intervention types. We propose that the issue may be even more concerning than the discrepancy between reported vs. coded COIs indicates, for at least three reasons. First, even when authors disclosed COIs, it was rarely the case that they reported all COIs we identified. An author may disclose that they receive royalties for a manual related to the intervention, but not also disclose that a clinic at their university offers the intervention to children.
Second, many of the most prevalent COIs we coded were seldom, if ever, disclosed in COI statements. Reporting was usually restricted to direct financial transactions, with far less attention to institutional affiliations and ideological commitments. For example, very few reports in which the researcher was also the intervention developer disclosed this as a COI, even if they reported other COIs. Similarly, none of the COI statements disclosed that a researcher had received speaker or workshop fees (except in one case, where the speaker fees reported by the author were not relevant to the intervention being studied). We were able to find evidence of paid workshops in some cases as these are often advertised online, but we were unable to locate any evidence (and were therefore unable to apply the relevant code) of researchers accepting speaking fees to discuss their intervention research. We think it very unlikely the case that, among the researchers who comprised the authors on the 150 reports in our study, none had been paid to speak about their intervention work. Likewise, none of the COI disclosure statements specified that researchers were employed by an entity that trains others to use the intervention being studied. This COI was also difficult to detect using online resources, as many academic programs do not specifically list curriculum details that would allow us to determine which early interventions are part of students’ training.
Third, COIs were difficult to detect if the research team and/or institution that they were affiliated with did not have a strong web presence. It was our impression that for research teams located outside the United States, the United Kingdom, Canada, and Australia there was less online information about their institution or professional activities. At present, it is unclear if web presence is correlated with having a conflict of interest (and it is therefore unclear whether and how geographic location of the research team contributes to this type of measurement error). Stronger guidelines for reporting COIs, applied internationally to outlets that publish autism early intervention research, would ensure that disclosure rates of COIs do not differ by geographic location.
We should note that under-reporting of COIs is not necessarily indicative of intentional deception on the part of researchers. Many journals do not require, or only recently began requiring, COI disclosures. Additionally, those that do require COI disclosures are often rather unspecific in terms of their definitions of the activities that constitute COIs. At present, we are not aware of any field-specific guidelines describing the types of COIs that autism early intervention researchers should report. As such, COIs that may have less clear or immediate financial benefits (such as being the developer of the intervention being tested, or working at an institution that trains others to use the intervention) seems to be considered by nearly all autism early intervention researchers as outside the purview of required reporting.
Finally, we think it is useful to note that the original meta-analysis did detect some evidence of publication bias (Sandbank et al., 2020), indicating that researchers (and/or journals) may not publish null results. This bias may be intertwined with the existence of COIs, and with the failure to report COIs. The net result is that current research may overestimate the effectiveness of some autism early intervention practices.
There are a few interesting patterns of COI occurrence by intervention type that could be examined in future research. First, interventions developed in fields that are likely to serve populations other than autistic children (e.g., cognitive behavior therapy, animal-assisted therapy), or that were developed several decades ago (e.g., TEACCH) may be more readily examined by researchers who were not intervention developers. Second, interventions that were designed in clinics appear to be especially likely to be studied by researchers who have some clinic affiliation or even employment in a clinic. Finally, it is worth noting that nearly 50% of developmental studies are conducted by researchers who appear to receive direct payments related to the intervention.
Influence of COIs on summary effect sizes
The series of meta-regressions we conducted did not detect significant differences between studies with no coded COIs, and studies with (a) any COI or (b) one of the eight categories of COIs we examined. However, we do think it is notable that the summary effect point estimate for studies with no coded COIs is lower than the summary effect point estimates for all other COI categories, even if not significantly so. When interpreting our results, it is important to consider that our method for coding COIs is susceptible to measurement error which would decrease the likelihood of detecting ‘true effects’ even if such effects exist (Shavelson & Webb, 1991). Given the under-reporting of COIs in this literature, and the fact that our coding method relied on our ability to find evidence of COIs (primarily through extensive web searches), the ‘no codable COI’ comparison category cannot be considered synonymous with a ‘no COI’ category. It is also possible that there were very few reports that were truly uninfluenced by any researcher COIs. Therefore, detecting significant effects of the various COIs we examined may be too high a bar given current reporting and research practices. Higher point estimates for summary effects calculated on studies with COIs as compared to point estimates for ‘no codable COIs’ is perhaps sufficient evidence to suggest the field begin to rethink COI reporting practices so that many more COIs are clearly disclosed within research reports.
Implications for future research
Requiring that authors report a wider variety of COIs than is currently common practice will have the immediate effect of increased transparency in autism early intervention research, which can help the field be more alert to interpreting research results in light of potential biases. Greater transparency in reporting COIs can go hand in hand with greater transparency in the conduct of research more generally. Pre-registration of trial protocols and open workflows that document study procedures could be required to ensure that researchers are making the best possible design decisions, and sticking to them throughout the research and dissemination process (Nosek, Spies, & Motyl, 2012).
Improved COI reporting practices will also enable future researchers to more adequately assess the influence of COIs on autism early intervention research findings (i.e., categorizing presence/absence and types of COIs will be less prone to measurement error). If it is found that particular types of COIs are associated with inflated effect sizes, precautions can then be taken to minimize these specific impacts, either in primary studies before they are conducted or afterwards with statistical correction. For example, given the documented evidence in other fields that studies in which the researcher is also the developer of the intervention are prone to inflated effect sizes, greater care could be taken to separate intervention developers, or any researcher with a stake in the success of an intervention (which includes prospects for career advancement) from the data collection and analysis process. These precautions could be specifically disclosed in research reports (e.g., as done in Tellegen et al., 2014). Additionally, grant funders could prioritize completely independent replications of promising interventions, so that efficacy can be documented when the developer has no involvement with the research.
Finally, subsequent research into the relative impacts of COIs could offer empirical guidance for estimating the differential and accumulated COI bias for a given summary effect. This would pave the way for statistical corrections akin to those used for publication bias, unreliable measures, and selection effects (Hunter & Schmidt, 2004). In addition to increasing the precision of summary estimates, thereby allowing researchers to better understand intervention effects, such correction would also reduce the likelihood of spurious moderator identification in meta-regression analyses, which is increased by the artificial inflation of between-studies heterogeneity introduced by COIs. In this way, increased reporting of COIs will enable autism early intervention researchers and clinicians to improve the accuracy of their intervention recommendations, both in regards to approach, as well as potential intervention attributes (e.g., dosage, setting, implementer) that may moderate intervention effectiveness.
In general, the few COIs that are reported in autism early intervention research are restricted to a subset of financial transactions, such as royalties received from the sale of manuals and assessments, or commercialized products used in the intervention (e.g., iPad Apps). It is unclear why researchers do not regularly report paid consultancies or speaking engagements, which also involve direct financial transactions, although this lack of reporting is consistent with other fields in psychology (Chivers, 2019). Autism researchers should consider broadening their conceptualizations of what constitutes COIs, and also include relationships such as affiliations with clinics that provide the intervention they study, or affiliations with institutions that train students to use the intervention. Both of these examples could result in researcher benefits such as salary support, and can reflect ideological commitments to particular intervention practices (Gorman, 2018). Perhaps even more important is that journal editors more specifically state an expectation that these relationships, affiliations, and commitments be explicitly reported in COI disclosure statements.
Finally, this study focused on COIs in relation to autism early intervention research, but there may be other areas of autism research where COIs may be in play. Notably, the use of diagnostic, screening, and assessment instruments by researchers who hold the patents to, and receive royalties from, the sale of those instruments is a COI that is likely pervasive across research topics.
Conclusion
This study is the first systematic investigation into COIs in autism early intervention research. We found that COIs exist in a majority of studies, but are widely unreported. While we are unable to provide strong evidence that COIs are associated with inflated effect sizes, improved reporting practices are necessary for researcher transparency and for enabling robust inquiry into the influence of COIs on autism early intervention research. Improved reporting of COIs may work hand-in-hand with efforts to improve study quality in autism early intervention research. If this is the case, researchers and clinicians will be able to offer more accurate information to families who are navigating the variety of early interventions that are currently offered.
Supplementary Material
Key points.
Conflicts of interest (COIs) have not previously been examined in autism early intervention research, but could potentially inflate effect sizes
In the context of a comprehensive meta-analysis, we coded all available group design, non-pharmacological autism early intervention studies for presence, reporting, and type of COIs
COIs were coded in 70% of studies, but fully reported in less than 6% of studies
We did not find robust evidence that COIs inflate effect sizes, but better reporting practices are needed to adequately determine whether and which types of COIs influence research findings.
Acknowledgements
The authors would like to thank Margaret Cassidy, Kacie Dunham, Jacob Feldman, Jenna Crank, Susanne Albarran, Sweeya Raj, and Prachy Mahbub for their work in screening and coding studies. K.B-B. has previously received fees for consulting with school districts on intervention practices, and teaches coursework on a range of intervention practices including traditional behavioral interventions, naturalistic developmental behavioral interventions (NDBIs), and TEACCH. S.C. was formerly affiliated with an entity that trained students to become Board Certified Behavior Analysts and provided Early Intensive Behavioral Intervention. M.S. directs a program that provides coursework approved by the Behavior Analysis Certification Board, and teaches courses on traditional behavioral intervention approaches and NDBIs. T.G.W. has previously received payment to provide both traditional behavioral and naturalistic developmental behavioral interventions. She is also employed in a department that received a training grant which will fund students seeking MS-SLP and BACB licensing. Finally, she has been funded via NIH and other internal and external funding agencies on projects testing the efficacy of several types of treatment, including NDBIs (e.g., ImPACT), as well as sensory-based and technology based interventions.
Footnotes
Conflict of interest statement: See Acknowledgements for full disclosures.
Supporting information
Additional supporting information may be found online in the Supporting Information section at the end of the article:
Appendix S1. List of studies included in the meta-analysis.
References
- Borenstein M, Hedges LV, Higgins JPT, & Rothstein HR (2009). Introduction to meta-analysis. West Sussex, UK: Wiley. [Google Scholar]
- Cherla DV, Viso CP, Holihan JL, Bernardi K, Moses ML, Mueck KM, … & Adams SD (2019). The effect of financial conflict of interest, disclosure status, and relevance on medical research from the United States. Journal of General Internal Medicine, 34(3), 429–434. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Chivers T (2019). Does psychology have a conflict of interest problem? Nature, 571, 20–23. [DOI] [PubMed] [Google Scholar]
- Committee on Publication Ethics (2019). Conflicts of interest/Competing interests. Retrieved from https://publicationethics.org/competinginterests
- Dawson M (2006, December 5). A motion against autism [blog post]. https://autismcrisis.blogspot.com/search?q=conflicts+of+interest
- Dawson M [@autismcrisis]. (2015, October 6). Meanwhile, autism behavior analysts with conflicts of interest routinely disclose no COIs, by omission or commission http://bit.ly/1PfTBLg [Tweet]. Twitter. https://twitter.com/autismcrisis/status/651546797064720384
- Delgado AF, & Delgado AF (2017). The association of funding source on effect size in randomized controlled trials: 2013–2015 - a cross-sectional survey and meta-analysis. Trials, 18, (125). [DOI] [PMC free article] [PubMed] [Google Scholar]
- Eisner M (2009). No effect in independent prevention trials: Can we reject the cynical view? Journal of Experimental Criminology, 5, 163–183. [Google Scholar]
- Eisner M, Humphreys DK, Wilson P, Gardner F (2015). Disclosure of financial conflicts of interests in interventions to improve child psychosocial health: A cross-sectional study. PLoS ONE, 10(11), e0142803 10.1371/journal.pone.0142803 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Gandhi AG, Murphy-Graham E, Petrosino A, Chrismer SS, & Weiss CH (2007). The devil is in the details: examining the evidence for “proven” school-based drug abuse prevention programs. Evaluation Review, 31(1), 43–74. [DOI] [PubMed] [Google Scholar]
- Gorman DM (2003). Alcohol & drug abuse: the best of practices, the worst of practices: the making of science-based primary prevention programs. Psychiatric Services, 54(8), 1087–1089. [DOI] [PubMed] [Google Scholar]
- Gorman DM (2005). Drug and violence prevention: rediscovering the critical rational dimension of evaluation research. Journal of Experimental Criminology, 1(1), 39–62. [Google Scholar]
- Gorman DM (2018). Can we trust positive findings of intervention research? The role of conflict of interest. Prevention Science, 19(3), 295–305. [DOI] [PubMed] [Google Scholar]
- Gorman DM, & Conde E (2007). Conflict of interest in the evaluation and dissemination of “model” school-based drug and violence prevention programs. Evaluation and Program Planning, 30, 422–429. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Goozner M, Caplan A, Moreno J, Kramer BS, Babor TF, & Husser WC (2009). A common standard for conflict of interest disclosure in addiction journals. Addiction, 104(11), 1779–1784. [DOI] [PubMed] [Google Scholar]
- Hedges LV, Tipton E, & Johnson MC (2010). Robust variance estimation in meta‐regression with dependent effect size estimates. Research Synthesis Methods, 1(1), 39–65. [DOI] [PubMed] [Google Scholar]
- Hunter JE, & Schmidt FL (2004). Methods of meta-analysis: Correcting error and bias in research findings. Thousand Oaks, CA: Sage. [Google Scholar]
- Institute of Medicine. (2002). Integrity in scientific research: Creating an environment that promotes responsible conduct. Washington, DC: National Academies Press. [PubMed] [Google Scholar]
- Ioannides JPA, Munafo JR, Fusar-Poli P, Nosek BA, & David SP (2014). Publication and other reporting biases in cognitive sciences: Detection prevalence, and prevention. Trends in Cognitive Sciences, 18, 235–241. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Nosek BA, Spies JR, & Motyl M (2012). Scientific utopia: II. Restructuring incentives and practices to promote truth over publishability. Perspectives on Psychological Science, 7(6), 615–631. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Okike K, Kocher MS, Wei EX, Mehlman CT, & Bhandari M (2009). Accuracy of conflict-of-interest disclosures reported by physicians. New England Journal of Medicine, 361(15), 1466–1474. [DOI] [PubMed] [Google Scholar]
- Petrosino A, & Soydan H (2005). The impact of program developers as evaluators on criminal recidivism: results from meta-analyses of experimental and quasi-experimental research. Journal of Experimental Criminology, 1(4), 435–450 [Google Scholar]
- Presnall N, Wester-Stratton CH, Constantino JN (2014). Parent training: Equivalent improvement in externalizing behavior for children with and without familial risk. Journal of the American Academy of Child and Adolescent Psychiatry, 53(8), 879–887. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Relman AS (1984). Dealing with conflicts of interest. New England Journal of Medicine, 310, 1182–1183. [DOI] [PubMed] [Google Scholar]
- Romain PL (2015). Conflicts of interest in research: Looking out for number one means keeping the primary interest front and center. Current Reviews in Musculoskeletal Medicine, 8, 122–127. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Sandbank M, Bottema-Beutel K, Crowley S, Cassidy M, Dunham K, Feldman JI, Crank J, Albarran S, Raj S, Mahbub P, & Woynaroski TG (2020). Project AIM: Autism intervention meta-analysis for studies of young children. Psychological Bulletin. [DOI] [PMC free article] [PubMed]
- Shavelson RJ, & Webb NM (1991). Measurement methods for the social sciences series, Vol. 1. Generalizability theory: A primer. Thousand Oaks, CA, US: Sage Publications, Inc. [Google Scholar]
- StataCorp (2017). Stata Statistical Software: Release 15. College Station, TX: StataCorp LLC [Google Scholar]
- Tellegen CL, & Sanders MR (2014). A randomized controlled trial evaluating a brief parenting program with children with autism spectrum disorders. Journal of Consulting and Clinical Psychology, 82(6), 1193–1200. [DOI] [PubMed] [Google Scholar]
- Wallace B, Small K, Brodley CE, Lau J, & Trikalinos TA (2012). Deploying an interactive machine learning system in an evidence-based practice center: abstrackr. In Proceedings of the 2nd ACM SIGHIT International Health Informatics Symposium, 819–824. [Google Scholar]
- Wells EM (2017). Evidence regarding the impact of conflicts of interest on environmental and occupational health research. Current Environmental Health Reports, 4(2), 109–118. [DOI] [PubMed] [Google Scholar]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.