Abstract
Introduction
Previous analysis of registered clinical trials has found a number of protocols result in changes in the registered primary outcome measures. This investigation determined if reported primary outcomes in chiropractic-related clinical trials registered in clinicaltrials.gov match their published results. Additionally, we assessed secondary outcomes, publication status and whether raw data were posted to the registry.
Methods
Clinicaltrials.gov was searched for chiropractic-related trials and having a completed status. If the study was published, outcome measures were compared between the clinicaltrials.gov entry and the published paper to assess for consistency.
Results
Within clinicaltrials.gov 171 chiropracticrelated protocols were identified with 102 of those published (59.6% publication rate). Ninety-two of the published papers (90.2%) had agreement between their primary outcome and the entry on clinicaltrials.gov and 82 (80.4%) agreed with the secondary outcomes.
Conclusion
A modest rate of agreement between clinicaltrials.gov entries and the published papers was found. While chiropractic-related clinical trials are fewer compared to medical trials, chiropractic-related research has a substantially better rate of primary and secondary outcome concordance with registered protocols.
Keywords: chiropractic, bias, clinical trial registry, research report, status of the profession
Keywords: MOTS CLÉS: chiropratique, biais, registre d’essais cliniques, rapport de recherche, état de la profession
Abstract
Introduction
En examinant des essais cliniques enregistrés, on s’est rendu compte qu’un certain nombre de protocoles faisaient varier les résultats principaux. On a mené une étude pour savoir si les résultats primaires d’essais cliniques sur la chiropratique enregistrés sur clinicaltrials.gov correspondaient à ceux publiés. On a aussi examiné les résultats secondaires, l’état de publication et cherché à savoir si les données brutes étaient publiées dans le registre.
Méthodologie
Dans la base de données Clinicaltrials.gov, on a repéré des essais cliniques sur la chiropratique qui étaient terminés. Lorsque l’essai clinique avait été publié, on a comparé les résultats au moment de son enregistrement sur clinicaltrials.gov à ceux parus dans des publications pour savoir s’ils concordaient.
Résultats
Sur le site clinicaltrials.gov, on a trouvé 171 études sur la chiropratique, dont 102 avaient été publiés (taux de publication:59,6 %). Pour quatrevingt-douze publications (90,2 %), on a observé une concordance entre les résultats primaires au moment de l’enregistrement sur clinicaltrials.gov et 82 (80,4 %) et les résultats secondaires.
Conclusion
On a observé un taux modeste de concordance entre les données à l’enregistrement sur clinicaltrails.gov et les données publiées. Les essais cliniques sur la chiropratique sont moins nombreux que des essais cliniques de médicaments. Mais le taux de concordance entre les résultats primaires et les résultats secondaires était considérablement plus élevé lorsque les protocoles de recherches sur la chiropratique sont enregistrés.
Introduction
Clinical trial registries are an important tool in evidence-based medicine to monitor selective reporting of outcome measures, publication bias, and duplication of trials. In addition, they establish records of non-published trials for clinicians and researchers who are interested in investigating similar hypotheses. Trial registration is required in the United States via the Food and Drug Administration Modernization Act1 and the Food and Drug Administration Amendments Act2. Biomedical journals that subscribe to publication standards of the International Committee of Medical Journal Editors (ICMJE)3 require that clinical trials are registered. Finally, the World Health Organization4 also requires trial registration.
In an analysis of registered clinical trials in medical research, Fleminger and Goldacre5 have found that a disappointing number of study protocols resulted in publications which change the registered a priori primary outcome (34.1% of registered trials). Likewise, there is a disappointing rate of unpublished trials. Huić, Marušić, and Marušić6 evaluated randomized control trial (RCT) completeness and agreement between clinicaltrials.gov and ICMJE publications which found comparable findings (38.8% of registered trials had changes to the “Primary outcome” field) to Fleminger and Goldacre5. Similar research on changing primary outcome measures has not been published on chiropractic-related studies, that we are aware of. A recent study by Wells and Lawrence7 evaluating bias in chiropractic and spinal manipulation research reported a “skew in favor of the intervention” (64.6%) of completed entries on clinicaltrials.gov. In addition, they noted that only 17.7% of completed entries on clinicaltrials.gov had results posted.
The primary purpose of this investigation was to determine if reported primary outcomes in chiropractic-related clinical trials registered in clinicaltrials.gov match their published results. Secondarily, other outcome measures and publication status are assessed.
Methods
Clinicaltrials.gov was searched for chiropractic-related trials. We identified a chiropractic-related trial exclusively by using the search terms “chiropractic”, “chiropractor”. As we were determining the concordance of registered trial to published manuscript, we also searched for completed status, as incomplete studies would not have been published, except as protocol papers. Publication status was determined by searching PubMed (pubmed.gov), Index to Chiropractic Literature (chiroindex.org), and Google Scholar (scholar.google.com) through 29 May 2020. Search terms for these included the official trial name, the entry name on clinicaltrials.gov, the National Clinical Trial (NCT) identifier, or searching for the principal investigator (as an author [au] search). If the study was published, the paper was acquired and outcome measures were compared between the clinicaltrials.gov entry and the published paper to assess for consistency in methods/tools used and time-frame of collection which was determined by two investigators independently. If disagreements among investigators could not be resolved through consensus discussion, a third investigator was consulted.
Results
Within clinicaltrials.gov 171 chiropractic-related protocols were identified. Of these protocols, 160 novel entries were found with the search term “chiropractic” and 11 novel entries were found with “chiropractor”. Twenty-five had results posted on clinicaltrials.gov and 102 were published. Twenty-nine of those entries produced multiple papers consisting of pilot studies, feasibility studies, protocol papers, clinical trials, mixed-methods trials, and poster presentations. Table 1 shows by year, the number of protocols on Clinicaltrials.gov that were registered and published. The vast majority of published studies, 93, were found by searching PubMed. The remaining nine studies were found using Index to Chiropractic Literature or Google Scholar.
Table 1.
Numbers of registered and published Chiropractic ClinicalTrials.gov Entries
| Year | Number of protocols registered year | Number of studies published per year |
|---|---|---|
| 2001 | 3 | 0 |
| 2002 | 1 | 1 |
| 2003 | 2 | 0 |
| 2004 | 1 | 1 |
| 2005 | 8 | 1 |
| 2006 | 20 | 0 |
| 2007 | 22 | 2 |
| 2008 | 17 | 6 |
| 2009 | 13 | 7 |
| 2010 | 10 | 13 |
| 2011 | 7 | 4 |
| 2012 | 13 | 7 |
| 2013 | 12 | 6 |
| 2014 | 10 | 12 |
| 2015 | 9 | 12 |
| 2016 | 8 | 12 |
| 2017 | 5 | 7 |
| 2018 | 6 | 9 |
| 2019 | 4 | 1 |
| 2020 | 0 | 1 |
| Totals | 171 | 102 |
Of the 102 studies published, 92 (90.2%) had agreement between their primary outcome and the listed entry on clinicaltrials.gov and 82 (80.4%) agreed with the secondary outcomes. Most published papers had matching outcome language compared to the clinicaltrial.gov entry counterpart and were counted as in agreement. For those that disagreed in primary outcome measures, four were due to improper entry of information on the clinicatrials.gov website; these included: no outcome measures reported, entering the outcome measure in the introduction information and not the Primary Outcome Measure field, and putting study design material in the Primary Outcome Measure field. The remaining six entries that were in disagreement had either changed primary outcome measurement tools, changed the timeframe of assessment, or omitted their stated outcome measures as per the clinicaltrials.gov entry.
Completed trials on clinicaltrials.gov had a 59.6% (102/171) publication rate and a 14.6% (25/171) rate of displaying their results on the website. Eighty-four of those published were from 2016 and earlier and the remaining eighteen were after 2017.
Discussion
A modest rate of primary outcome agreement (90.2%) between clinicaltrails.gov entries and the published papers (59.6% publication rate) were found. This compares favorably to what Fleminger and Goldacre5 and Huić, Marušić, and Marušić6 reported, that 38.8% and 64.5% of RCTs in medical literature had discrepant primary and secondary outcomes, respectively. Ramagopalan et al.8 reported that 31.7% of registered interventional studies between 1999 and 2012 on clinicaltrials.gov had changed their primary outcome measure between the initial entry and obtaining a completed status. In a revision to the previous study, Ramagopalan et al.9 looked at completed interventional studies between 1999 and 2014 that had results published on clinicaltrials.gov and reported that 92.5% of those studies changed their primary outcome measure between initial entry and obtaining a completed status. While these two papers do not assess agreement between the clincaltrials.gov entry and a published paper, it does demonstrate a large proportion of them were not consistent with their initial goal. The authors of those papers attributed this to industry funding and reporting of statistically significant outcomes. Fleminger and Goldacre5 reported a 44.4% publication rate of trials registered in clinicaltrials.gov and the European Union Clinical Trials Register.
Compared to what we found in the literature, chiropractic-related human subjects studies show a better primary outcome agreement and publication rate. Why chiropractic research has a better agreement and publication rate than the biomedical research community is not explained in the data. We believe that this may be due to two factors. A significant research effort in the chiropractic profession is relatively recent10–13 and thus likely to have learned from advances made in the overall biomedical research community. For a so-called marginal profession14 there is an imperative to do better than the overall community. Additionally, there is not as large an amount of research funding provided to the chiropractic profession, as compared to the wider medical community (i.e., from pharmaceutical or medical equipment companies). Because of this, there is not as much financial pressure to produce statistically significant outcomes.
Wells and Lawrence7 found bias in chiropractic-related publications and spoke to a need for more investigators to add results to clinical trial registries. Our study found more “chiropractic” studies than Wells and Lawrence7 when searching the same database (160 vs. 65). Wells and Lawrence searched for the terms “chiropractic” (data collection ended in Aug 18) and “spinal manipulation” (data collection ended in May 19), while we searched for “chiropractic” and “chiropractor”. Additionally, they searched for all registered protocols, not just completed entries. The data collection time periods and different search methods may account for the differences in entities found.
Unfortunately, only 14.6% (25/171) studies had included results on clinicaltrials.gov. One purpose of clinicaltrials. gov is to have a database of results from clinical trials that have and have not been published, so without posted results from these protocols, the information they obtained is lost. This leads to more publication bias and loss of clinical information. It is important to note, 13 of the unpublished entries on clinicaltrials.gov were recent (within the last three years) and may be still seeking publication at the time of this investigation.
A limitation of our study could relate to the search terms used. While using “chiropractic” or “chiropractor” should include the majority of chiropractic-related clinical trials, having more inclusive search terms, such as “spinal manipulation” may have yielded more results.
Conclusion
While chiropractic-related clinical trials are fewer in number compared to medical trials, chiropractic-related research has a substantially better rate of primary and secondary outcome concordance with registered protocols and a better publication rate. A possible explanation for this is that funding for chiropractic studies is comparatively sparse and authors of these studies are ambitious to report findings, whether positive or negative. They likely do this in an attempt to reduce bias and provide evidence on treatment effectiveness or ineffectiveness.
To date, we found one previous study7 that has evaluated chiropractic’s publication rate and no chiropractic studies assessing outcome agreement in a clinical trial registry. As other authors5–9 involved in medical research have reported, completion status and errors in registry information is common in healthcare research; it is encouraging that we found moderately good agreement in outcome measures with registered protocols. To further build upon the knowledge base of manual health care, chiropractic investigators need to continue to complete studies, upload results to clinical trial registries, and seek publication regardless of the study findings. In general, it is important to publish negative results so as not to introduce positive bias into meta-analyses. Both positive and negative findings are important when evaluating treatments and determining the best care for patients. Additionally, the NCT identifier should be included in published papers to better link with the clinical trial registry.
With regards to our investigation and future investigations; future studies should strive to include all chiropractic-related search terms to ensure proper and full representation of chiropractic clinical research.
Footnotes
The authors have no disclaimers, competing interests, or sources of support or funding to report in the preparation of this manuscript.
References
- 1.Food and Drug Administration Modernization Act of 1997. [Oct 15, 2019]. Public Law 105-115– Nov 21, 1997. Retrieved from https://www.govinfo.gov/content/pkg/PLAW-105publ115/pdf/PLAW-105publ115.pdf#page=16.
- 2.Food and Drug Administration Amendments Act of 2007. [Oct 15, 2019]. Public Law 110-85–Sept 27, 2007. Retrieved from: https://www.govinfo.gov/content/pkg/PLAW-110publ85/pdf/PLAW-110publ85.pdf#page=82.
- 3.Clinical Trial Registration. International Committee of Medical Journal Editors; [Oct 15, 2019]. Retrieved from: http://www.icmje.org/recommendations/browse/publishing-and-editorial-issues/clinical-trial-registration.html. [Google Scholar]
- 4.Organisation WH; International Standards for Clinical Trial Registration. 2012 [Google Scholar]
- 5.Fleminger J, Goldacre B. Prevalence of clinical trial status discrepancies: A cross-sectional study of 10,492 trials registered on both ClinicalTrials.gov and the European Union Clinical Trials Register. PLoS ONE. 2018;13(3):e0193088. doi: 10.1371/journal.pone.0193088. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 6.Huić M, Marušić M, Marušić A. Completeness and changes in registered data and reporting bias of randomized controlled trials in ICMJE journals after trial registration policy. PLoS ONE. 2011;6(9):e25258. doi: 10.1371/journal.pone.0025258. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Wells B, Lawrence D. Potential publication bias in chiropractic clinicaltrials.gov. J Can Chiropr Assoc. 2020;64(1):82–87. [PMC free article] [PubMed] [Google Scholar]
- 8.Ramagopalan S, Skingsley AP, Handunnetthi L, et al. Prevalence of primary outcome changes in clinical trials registered on ClinicalTrials.gov: a cross-sectional study. F1000Res. 2014;3:77. doi: 10.12688/f1000research.3784.1. Published 2014 Mar 26. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9.Ramagopalan SV, Skingsley AP, Handunnetthi L, et al. Funding source and primary outcome changes in clinical trials registered on ClinicalTrials.gov are associated with the reporting of a statistically significant primary outcome: a cross-sectional study [version 2; peer review: 2 approved] F1000Research. 2015;4:80. doi: 10.12688/f1000research.6312.2. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 10.Meeker WC, Haldeman S. Chiropractic: a profession at the crossroads of mainstream and alternative medicine. Ann Intern Med. 2002;136(3):216–227. doi: 10.7326/0003-4819-136-3-200202050-00010. [DOI] [PubMed] [Google Scholar]
- 11.Lawrence DJ, Meeker WC. Commentary: the National Workshop to Develop the Chiropractic Research Agenda: 10 years on, a new set of white papers. J Manipulative Physiol Ther. 2006;29(9):690–694. doi: 10.1016/j.jmpt.2006.09.004. [DOI] [PubMed] [Google Scholar]
- 12.Haas M, Bronfort G, Evans RL. Chiropractic clinical research: progress and recommendations. J Manipulative Physiol Ther. 2006;29(9):695–706. doi: 10.1016/j.jmpt.2006.09.005. [DOI] [PubMed] [Google Scholar]
- 13.Triano JJ, Goertz C, Weeks J, Murphy DR, Kranz KC, McClelland GC, et al. Chiropractic in North America: toward a strategic plan for professional renewal--outcomes from the 2006 Chiropractic Strategic Planning Conference. J Manipulative Physiol Ther. 2010;33(5):395–405. doi: 10.1016/j.jmpt.2010.05.002. [DOI] [PubMed] [Google Scholar]
- 14.Saks M. Inequalities, marginality and the professions. Current Sociol. 2015;63(6):850–868. [Google Scholar]