The research enterprise at Canadian Memorial Chiropractic College

Abstract

Objectives

A bibliometric survey was conducted, using network and textual analysis tools, to assess the current state of the research enterprise at Canadian Memorial Chiropractic College and to augment planning processes.

Methods

Searches were conducted via several databases to identify publications attributable to the institution. Bibliometric data were summarized and post-processed using the programme VosViewer and analysis tools provided in the Web of Science.

Results

Canadian Memorial Chiropractic College is a productive source of peer-reviewed publications supported by a diverse suite of funding agencies and collaborating institutions, and published across a broad range of journals.

Conclusions

As a private, single-purpose educational institution, awarding a qualification only in chiropractic, Canadian Memorial Chiropractic College probably performs well in its class of institution in terms of research productivity. However, assessment is constrained by inconsistencies on the part of authors, journals and databases in archiving data.

Introduction

On the occasion of the 75^th anniversary of the founding of Canadian Memorial Chiropractic College (CMCC), members of the research faculty felt it useful to review the research enterprise of the institution with reference to the institution’s current research agenda. In 2016, Canadian Memorial Chiropractic College had revised its research agenda and identified five core research streams: i) Health and Wellness; ii) Biological Basis of Musculoskeletal Injury and Manual Therapies; iii) Clinical and Health Services Research; iv) Education in Healthcare; v) Knowledge Translation and Health Policy.¹

At the time that the researchers met to formulate the proposed research themes, no comprehensive quantitative information was presented concerning the current research capacity or output of the institution. In fact, while it appears that Canadian chiropractors value chiropractic research², and CMCC-affiliated chiropractors have taken a leadership role in promoting chiropractic research in Canada (see ³), the peer-reviewed literature on the Canadian chiropractic research enterprise is all but non-existent (but see 4,5). Thus, each researcher brought with them their personal perceptions of the place and trajectory of CMCC within the broader enterprise of research of relevance to chiropractic. It has previously been argued that an essential step in developing a research agenda for chiropractic in Canada would be to take stock of current resources.⁶ The above-referenced study by Stuber et al.⁵ of Canadian chiropractic research had painted a picture of modest research capacity, but used anonymized surveys such that it was not possible to parse out data specific to CMCC. With regard to the research streams identified by CMCC researchers as priorities, there was some congruence with recommendations of a previous Canadian research agenda workshop which had advocated for a focus on spinal pain, but which made no mention of health services research, nor research concerning knowledge translation and health policy.⁶ In summary, the process of developing research agendas for chiropractic in Canada, and at CMCC in particular, has not adhered to any standard, and has been of unproven value in identifying and serving the needs of stakeholders. Under these circumstances, while celebrating the substantial research output of the institution, it seems appropriate to support enthusiasm with objective measures, and to augment the current planning processes with the best available information.

Therefore, the current study served to collate objective, quantitative information on the research capacity of Canadian Memorial Chiropractic College and its current scholarly output, and to determine congruence with the needs of the five research streams identified by the researchers. No hypotheses were tested.

Methods

Extending the aforementioned Canada-wide study of research conducted between 2012 and 2017⁴, records of research publications for the years 2012 to 2020, inclusive, were obtained from the Office of Research Administration at CMCC. Prior to 2012, in-house records of publications were not archived systematically. Additionally, literature searches were conducted in January 2021 through Web of Science (WoS), PubMed and the Index to Chiropractic Literature (ICL) to identify all primary data peer reviewed articles attributed to Canadian Memorial Chiropractic College from December 31, 2020 back to first citations in WoS and PubMed, and back to January 2001 in ICL.

In more detail, a WoS search was conducted on January 17, 2021 using the search string ZP=M2H3J1 OR ZP=M2H 3J1 or OO=Can* Mem* Chir* Coll*, Time-span= All years. Indexes=SCI-EXPANDED, SSCI, A&HCI, CPCI-S, CPCI-SSH, ESCI. Similarly, a search of PubMed was conducted on January 21, 2021 using the search string: (“Canadian Memorial Chiropractic College”[Affiliation]) AND (“1900/01/01”[Date - Publication]: “2020/12/31”[Date - Publication]). ICL did not permit searching by affiliation, and so all fields were searched for either Canadian Memorial Chiropractic College OR CMCC. The results of all searches were combined, duplicates were eliminated, as were editorials, commentaries and letters to the editor to produce a database referred to hereinafter as PubMed-supplemented. From each article in this database, journal name, funding agencies, collaborating institutions and co-authors were also recorded and correlated. Despite the smaller number of citations in WoS, the utilities in this resource were also used for certain analyses as set out below.

Thus, from the PubMed-supplemented database, numbers of publications per year and publications per journal were calculated. Numbers of acknowledgements of funding agencies were also calculated.

Citations per article for the years 2001 to 2020 were obtained from WoS for the 100 most highly cited articles out of 508 publications listed in total for this period. For the 20 most cited articles, numbers of citations were correlated with the corresponding authors. PubMed and ICL did not provide tracking of citations by affiliated institution.

In order to map the interactions of CMCC affiliated authors, WoS full records for CMCC-attributed papers were entered directly into the network mapping software, Vos-Viewer V1.6.16.⁷ Co-authorships were mapped for all authors with a minimum of three publications, an arbitrary threshold to reduce cluttering on co-authorship maps.

In order to derive leading research themes, two approaches were taken. In the first instance, titles and abstracts from WoS were analyzed with VosViewer to visualize linkages in keywords which occurred more than five times in the entire corpus. For this analysis, Vos-Viewer automatically eliminates navigational terms such as ‘purpose’ and ‘methods’, plus function words including articles (‘the’, ‘a’, ‘an’) and prepositions (‘in’, ‘on’, etc.). Additionally, the theme function in WoS was used to automatically assign thematic areas to articles.

Results

The search of PubMed from inception to 2020 yielded 643 articles, whereas the search of WoS for the same period yielded 625 articles, while the ICL search identified 237 articles. Number of publications per year for the WOS and PubMed searches are plotted in Figure 1, as are the numbers of publications for 2012 to 2020 per ICL and PubMed-supplemented. Primary data publications, systematic reviews and meta-analyses were identified by reviewing each citation individually to eliminate editorials, commentaries and letters to the editor.

Figure 1.

Total number of publications per year by database

For the 99 journals associated with the 456 CMCC-affiliated articles published from 2012 to 2020 (per PubMed⁺), 52 carried only a single publication. The Journal of the Canadian Chiropractic Association was, by far, the most popular venue of publication, with 129 articles, i.e. 28% of the total for this period (Figure 2).

Figure 2.

Total number of publications per journal for the period 2012 to 2020 per PubMed-supplemented

The numbers of citations per year for the 100 most cited CMCC articles according to WoS (PubMed and ICL do not provide this function) are shown in Figure 3. This WoS utility did not distinguish between publication type, but we note that in the period of 2001–2020 there were only six editorials, eight letters to the editor and eight commentaries attributed to CMCC. Hence, far and away the majority of citations would be associated with primary data articles, systematic reviews and meta-analyses.

Figure 3.

Citations per year per WoS for top 100 cited articles

With regard to funding agencies, both PubMed-supplemented and WoS records indicated that the CIHR was the agency most frequently cited despite the different time ranges for these databases (Figure 4). Of the 432 acknowledgements in PubMed-supplemented articles from 2012 to 2020, 292 agencies were cited only once. It is worth noting that the corresponding authors for the most cited article and three others of the 20 most cited articles have since retired from CMCC.

Figure 4.

Funding institutions as percentage of acknowledgements in PubMed-supplemented (from 2012 to 2020) and WoS (over all time).

Discerning collaborating institutions was problematic as different authors would cite the same institution in different ways or would acknowledge departments or programs rather than the institutions which housed them. Nonetheless, the most commonly acknowledged collaborating institutions according to WoS and PubMed-supplemented were the University of Toronto, McMaster University, the Institute of Work and Health (Ontario), the University of Alberta and Ontario Tech University (formerly UOIT).

With regard to collaborations, VosViewer was used to generate co-author networks for WoS output (over all time), with authors anonymized (Figure 5). Node size is scaled according to number of publications in WoS and proximity of nodes is scaled according to frequency of collaboration. An internal algorithm was used to identify colourised clusters, i.e. groups of authors who more frequently collaborated with each other.

Figure 5.

Collaborative networks of anonymized authors per VosViewer based on WoS output over all time.

With regard to research themes, WoS used an internal algorithm to assign each article to one or more general research fields as shown in Table 1 below. The sum of assigned themes is greater than 100% because some articles were assigned to more than one theme.

Table 1.

Research fields assigned to publications retrieved through WoS.

Web of Science Categories	records	% of 625
Rehabilitation	297	47.52
Health care sciences services	217	34.72
Integrative complementary medicine	215	34.4
Orthopaedics	121	19.36
Clinical neurology	95	15.2
Sport sciences	53	8.48
Neurosciences	33	5.28
Rheumatology	32	5.12
Medicine general internal	29	4.64
Public environmental occupational health	19	3.04
Anesthesiology	18	2.88
Cell biology	16	2.56
Biochemistry molecular biology	13	2.08
Engineering biomedical	13	2.08
Physiology	13	2.08
Biology	12	1.92
Education scientific disciplines	9	1.44
Health policy services	9	1.44
Anatomy morphology	7	1.12
Ergonomics	7	1.12
Multidisciplinary sciences	7	1.12
Social issues	7	1.12
Engineering industrial	5	0.8
Psychology applied	5	0.8
Medicine research experimental	4	0.64

As a companion strategy, titles and abstracts of articles retrieved from WoS were analyzed using the text analysis function in VosViewer, counting each instance of any word that occurred more than five times in the entire corpus. For the sake of legibility, of the 528 words which achieved this threshold, 317 (60%) with the highest relevance, i.e. over-representation compared to general English, were mapped as in Figure 6. Node size is scaled to frequency of occurrence in the corpus, node-to-node proximity is scaled according to number of co-occurrences.

Figure 6.

Co-occurrence of keywords in titles and abstracts of articles retrieved through WoS over all time.

Discussion

The work conducted for this paper takes a new approach to creating a chiropractic research agenda: one where the institution becomes the primary locum for data collection, instead of the individual researcher. The literature supports looking beyond the researcher as a unit, and focusing instead on the environment in which they operate. Bland et al.⁸ found that individual factors such as age, gender and department type were not significant predictors of this type of research productivity. Rather, once an institution recruited individuals with a passion for research, the departmental environment best predicted how well researchers would fare through: non-monetary recognition of research, emphasis by leadership of institutional research mission, and productivity in grantsmanship. Looking more broadly at research environments also allowed us to incorporate larger and more complex information than individual researchers can provide. Since most institutions report internally on researcher activity, and co-authors must report an affiliation in order for manuscripts to appear in journals, it is believed that this approach will yield better data with which to construct an overview of research activity.

Stuber et al.⁵ identified two areas which were necessary for the creation of a research agenda for chiropractic: public consultation and an accounting of available research resources. Inventorying of researcher activity at our institution was intended to complement focus on the latter area, as we believe that it must first be understood where researchers are already active before consulting with patients and other stakeholders concerning future research aims.

The survey by Stuber et al.⁵ garnered a 7% response rate, and revealed 530 publications over five years, but included editorials and commentaries in their corpus. Our institutional level search more completely captured data through affiliation, and additionally allowed for the capture of metadata (e.g., whether research is funded, and by whom), which could then be collated with other data on file. While we recognize Stuber et al.’s assertion that there is a hierarchy in types of research, we continue to include case studies in our accounting as we recognize the generative contribution that case studies can make to clinical research.

For CMCC, the publications data show growth which ramped up quickly around 2012 (Figure 1). The peak in publications data in 2015 coincided with the culmination of the Ontario Protocol for Traffic Injury Management (OPTIMa) Collaboration. The sudden growth in publications, including those produced by OPTIMa, is rooted in the formation of the Ontario Tech University-CMCC Centre for Disability Prevention (CDPR) in 2012. The CDPR, which draws personnel and resources from both institutions, produces systematic reviews as its main publication output. The publications associated with OPTIMa number at least 28 to date, with 13 of those first appearing in print (either in full, or e-publication format) in 2015. However, if all publications with the CDPR affiliation are extracted from the dataset on CMCC publications, it is clear that productivity continued to climb in other areas of the institution until about 2016. Since then, if the outlier year 2015 is overlooked, growth in publications in all areas has steadied out to roughly 56 publications per year until 2020.

We diversified our analysis to look at indicators other than number of publications. Publication venue analysis has proved useful in internal evaluations at CMCC, since it aids in the determination of the audience researchers are reaching. We believe it is important to a research enterprise that the work it produces be viewed by the scientific community to invite criticism, and to foster growth in specialized areas of inquiry. CMCC has therefore tracked where its researchers publish manuscripts. Many different publishing venues have been utilized, each with the different areas of specialty in which CMCC researchers engage, such as biomechanical research, or basic sciences research on mechanisms of pain and injury, or health policy research. Thus, there has been a growth over time in publication venues where CMCC research appears. Indeed, CMCC’s Vision and Mission speak to creating leaders in spinal health, and to deliver world class chiropractic research.⁹ Nonetheless, it is also important to target journals, such as the Journal of the Canadian Chiropractic Association, where CMCC’s stakeholders form a great part of the readership. Therefore, it should be welcomed that a great number of CMCC publications appear in the JCCA, and, at the same time, researchers continue to expand their work to other journals, especially as CMCC recruits the next generation of researchers, who must grow in their own research specializations.

On a final note for publications data, funder acknowledgments and citations data were analyzed to determine the impact, and the potential for impact, that the institution’s researchers have on their areas of study.

The number of times CMCC-affiliated works are cited is also an indicator of how often the research produced by the institution is read, and then used to create other works. Note that our projections come from WoS, and are therefore more conservative estimates, since only about 63% of the publications we found from 2012–2020 are listed on that platform. The data provided in Figure 3 are for the top 100 cited articles only, and must therefore not be interpreted as cumulative. Nonetheless, there appears to be a continuous upward trend suggesting continuing growth of impact within the research community.

Researchers will almost certainly attest to the level of commitment that it takes to write an application for funding, whether for small, medium, or larger grants. Additionally, the competitive nature of requests for proposals necessarily includes critique by members of a discipline with a history of securing funding. Grantsmanship success, especially in a Canadian setting for a privately funded institution, is laudable, when considering the restrictive eligibility criteria and fierce competition for federal government grants, and the small pool of funding opportunities specific to the chiropractic profession. The standards for reporting whether a work has been funded vary by journal, as well as funder obligations. Therefore, our results may underreport research funding. It is stipulated in all funding contracts for Tri-Agency grants that the funding agent must be named in resulting publications. Therefore, it was expected that the Canadian Institutes of Health Research (CIHR) would be the most cited funding agency in publications with CMCC affiliations. Administrative records at CMCC show that from 2006–2020, 17% of grant applications logged were for funding from the federal government, with a success rate of 32%. Of note, one third (32%) of the applications logged were collaborative, where another institution was named as the sponsor. As a consequence, it is likely that the success rate is positively skewed, and that other collaborative applications were not captured.

With regard to the nature of collaborations involving CMCC-affiliated authors, it appears that, in comparison to Canada-wide communities of chiropractic-affiliated authors⁴, there is a greater cohesiveness. Researchers who more frequently collaborate within a particular group nonetheless commonly collaborate outside of their preferred group (Figure 5). They do not, to reluctantly use a hackneyed metaphor, ‘occupy silos.’ The factors behind this phenomenon remain to be explored, but might include the social environment of the institution, access to complementary researchers and resources, or perhaps a generalist rather than specialist bent to some researchers.

With regard to the thematic analysis of CMCC-associated research (Table 1, Figure 6), there would appear to be a bias towards the study of clinical and professional issues, and towards musculoskeletal, neural and immunological influences on health and disease. These foci are congruent with the CMCC core research streams of i) Biological Basis of Manual Therapies, ii) Clinical and Health Services Research and v) Knowledge Translation and Health Policy. The CMCC core research streams of iii) Education in Healthcare and iv) Health and Wellness appear to be underrepresented in actual research activity to date.

This paper did not attempt an analysis of the productivity of individual researchers, but rather was intended as an overview of the research enterprise of the entire institution. However, and recognizing the importance of personal characteristics in personal productivity, it is clear that institutional and leadership characteristics strongly influence the behaviour of individual researchers.^8,10 In this regard, a recent unpublished self-study facilitated by an external researcher/administrator found a high degree of satisfaction among faculty with physical resources and brokered opportunities, but a low level of satisfaction with the fostering of a research culture, intra-institutional communications and research leadership.

Limitations

The greatest amount of data used in this study was derived from electronic databases, each of which has its own strengths and weakness. ICL has a clear focus on chiropractic research, but is relatively under-resourced. PubMed and WoS are both well-resourced, but until fairly recently have not indexed many peer-reviewed chiropractic journals, which, themselves are a recent phenomenon. PubMed probably has the broader scope, but provides limited post-processing of bibliometric data, while WoS involves a more ‘curated’ suite of journals, but nonetheless provides informative analyses. Hence, the approach of this study of attempting to triangulate the true nature of the CMCC research enterprise through multiple lenses.

Additional challenges involved inconsistencies in the recording or formatting of information by authors, journals and databases. Hence, for example, authors would find multiple ways to record their own names, multiple ways to identify the same affiliation, and multiple ways to identify funders. Journals and databases were inconsistent concerning which bibliometric information they recorded, and which information was searchable/retrievable.

Beyond these technical challenges, the terms research productivity and research excellence have been the subject of continuous and methodical debate across the globe for half a century, especially since they have been used by institutions as instruments of educational reform in the past 20 years.¹¹ In 2005, Bland et al.⁸ synthesized decades of literature in an attempt to identify characteristics associated with faculty research productivity, but the definition for productivity inevitably came back to publications as the main output. The Bland et al. study data are now 20 years old, and relied on the perceptions of those surveyed to draw conclusions about what defined research productivity. Investigators have continued to diversify the search for predictive factors of research productivity using publications as currency.¹¹ However, the use of a small set of metrics on output to quantify success draws criticism about how this practice may in fact be damaging to the traditional values of research, and detrimental to activities that are necessary for its enrichment.¹² Therefore, our use of publication activity as a proxy for research productivity must include a disclaimer that we recognize this approach is flawed. Further discussion and research on ways to measure research productivity must be created and must speak to the creation of a culture where engagement in activities that promote the values of good research is merited. Agate et al.¹² propose that this can be done through measuring the use of citations in syllabi, and placing greater emphasis on recognizing peer review.

Notwithstanding these limitations, the authors hope that the current and relatively novel use of social network analysis and textual analysis of research activity advance our shared understanding of the chiropractic research enterprise.