Improving the quality of evidence production in rehabilitation

Abstract

The paper introduces the Special Sections of the European Journal of Physical and Rehabilitation Medicine dedicated to the 5^th Methodological Meeting of Cochrane Rehabilitation. It introduces Cochrane Rehabilitation; its vision, mission and goals; discusses why the Methodological Meetings were created; and reports on their organisation and previous outcomes. The core content of this editorial is the 5^th Methodological Meeting held in Milan in September 2023. The original title for this meeting was “The Rehabilitation Evidence Ecosystem: useful study designs.” The focus of the Milan meeting was informed by the lessons learned by Cochrane Rehabilitation in the past few years, by the new rehabilitation definition for research purposes, by the collaboration with the World Health Organization (WHO), and by the REH-COVER (Rehabilitation COVID-19 Evidence-Based Response) action. During the Meeting, participants discussed the current methodological evidence on the following: RCTs in rehabilitation coming from meta-epidemiological studies; observational study designs – specifically the IDEAL Framework (Idea, Development, Exploration, Assessment, Long-term study) and its potential implementation in rehabilitation and the Target Trial Emulation framework: Single Case Experimental Designs; complex intervention studies: health services research studies, and studies using qualitative approaches. The Meeting culminated in the development of a first version of a “road map” to navigate the evidence production in rehabilitation according to the previous discussions. The Special Sections’ papers present all topics discussed at the meeting, and a methodological paper about choosing the right research question, presenting final results and the “road map” for evidence production in rehabilitation.

This paper introduces the two Special Sections of the European Journal of Physical and Rehabilitation Medicine dedicated to the 5^th Methodological Meeting of Cochrane Rehabilitation. It introduces Cochrane Rehabilitation; its vision, mission and goals; discusses why the Methodological Meetings were created; and reports on their organisation and previous outcomes. The core content of this editorial is the 5^th Methodological Meeting held in Milan in September 2023. The original title of this meeting was “The Rehabilitation Evidence Ecosystem: useful study designs.” The focus of the meeting was informed by lessons learned by Cochrane Rehabilitation in the past few years, by the new rehabilitation definition for research purposes, by the collaboration with the World Health Organization (WHO), and by the REH-COVER (Rehabilitation COVID-19 Evidence-Based Response) action. The papers in the Special Sections present all topics discussed at the Meeting, and a methodological article which explores choosing the right research question, presenting final results and the“road map” for evidence production in rehabilitation.

Cochrane Rehabilitation

The vision of Cochrane Rehabilitation (https://rehabilitation.cochrane.org/about-and-contacts/about-us) is of a world where all clinical decisions in rehabilitation are based on the best available evidence that is easily accessible and understandable by all rehabilitation stakeholders. The mission of Cochrane Rehabilitation is to improve the production, synthesis, dissemination, implementation and impact of rehabilitation evidence at an individual, organisational and health policy level, with a global view beyond professions, culture, language and economic resources. Cochrane Rehabilitation aims to promote evidence-based clinical practice, combining research evidence with clinical expertise and patient values. Cochrane Rehabilitation Goals are to:

• create a global rehabilitation network connecting all stakeholders and individuals involved in producing, disseminating, and implementing evidence-based clinical practice;

• undertake knowledge translation of Cochrane reviews relevant to rehabilitation, with dissemination to all relevant audiences and intermediaries in different formats and languages;

• increase knowledge and skills to:

• implement evidence-based clinical practice in rehabilitation through education and training for stakeholders;

• produce and synthesise evidence relevant to rehabilitation;

• improve the methodology of production and synthesis of evidence relevant to rehabilitation;

• develop specific strategies with projects and activities:

• focusing on the priorities and needs of Low and Middle-Income Countries;

• to impact health policy;

• synthesise or guide to synthesise the best available evidence relevant to rehabilitation according to identified priorities and emergent needs.

Cochrane Rehabilitation Methodology Meetings

Cochrane Rehabilitation regularly convenes Methodology Meetings (CRMM) to gather experts, including according to the topic and needs clinical researchers from all rehabilitation professionals’ groups, methodologists, epidemiologists, policymakers and patients. Participants are invited according to the explored topics. Meeting objectives focus on research methodology relevant to rehabilitation and include:

• developing new ways of thinking about evidence gathering, evidence generation and use of evidence in rehabilitation;

• identifying methodological issues specific to rehabilitation research and how these might be addressed;

• exploring strategies to inform and educate stakeholders within the rehabilitation field about the different ways high quality, timely and relevant research can be created;

• informing people outside rehabilitation about the ways that rehabilitation research can and should be generated, disseminated, understood and used.

The CRMM organisation includes different stages:

• definition: identification of topics and contributors;

• preparation: the identified contributor’ teams produce preparatory papers which are distributed prior to the meeting to all participants;

• discussion: the in person meeting is held where the preliminary documents are presented and thoroughly discussed in plenary sessions; with smaller discussion groups organized according to the need;

• dissemination, including:

• finalisation of the discussed papers;

• dissemination of papers through publication of one or more Special Issues in a scientific journal; and

• presentation of results at various global and national scientific meetings.

The 1^st CRMM was held in collaboration with the International Society of Physical and Rehabilitation Medicine (ISPRM) in Paris (France) in 2018 and initiated an exploration of the methodological problems in the field. Results included ten papers published in a Special Section of the European Journal of Physical and Rehabilitation Medicine in 2019.¹ The 2^nd CRMM happened before the ISPRM 2019 meeting in Kobe (Japan) and culminated with six papers published in a Special Section of the American Journal of Physical Medicine and Rehabilitation in 2020. The meeting focused on specific research issues² and, most of all, launched the Randomised Controlled Trials Rehabilitation Checklist (RCTRACK) project.³ The 3^rd and 4^th CRMM happened in 2020, one in February in Milan (Italy) and the other in March, prior to ISPRM 2020 in Orlando (USA). The 3^rd CRMM focused on the rehabilitation definition with five papers published in a Special Section of the European Journal of Physical and Rehabilitation Medicine in 2020,⁴ culminating after another year of work in the publication of the new rehabilitation definition for research purposes co-published in 2022 by three journals.⁵ The 4^th CRMM was entirely dedicated to the RCTRACK project and resulted in one editorial co-published in three Special Sections by different journals that hosted the final twelve papers: the American Journal of Physical Medicine and Rehabilitation in 2020, and the European Journal of Physical and Rehabilitation Medicine and the Archives of Physical Medicine and Rehabilitation in 2021.⁶

The 5^th Cochrane Rehabilitation Methodological Meeting

The topic originally chosen for the 5^th CRMM in Milan in September 2023 was “The Rehabilitation Evidence Ecosystem: Useful Study Designs”. This topic was chosen because of the lessons learned by Cochrane Rehabilitation in the past six years, specifically from the work completed in three project areas: identification of the new rehabilitation definition; formal collaboration with the WHO, and the sustained response to the global pandemic emergency known as REH-COVER action.

The new rehabilitation definition for research purposes was developed with 80 global stakeholders representing most professional and stakeholder groups in the field.⁵ It was developed to address the need to clearly differentiate between what may be considered or not considered to be “rehabilitation.” This definition positions rehabilition as a complex intervention, and therefore the Medical Research Council guidance on the evaluation of complex interventions could be used as a frame of reference for rehabilitation research.⁷ The new definition changed the reference framework by providing a clarifying statement for rehabilitation research and it has consequences yet to be fully understood.

A second set of lessons came to Cochrane Rehabilitation while working with the WHO to identify the evidence to produce the Package of Interventions for Rehabilitation,⁸ a tool offered to countries as the minimum rehabilitation services needed to provide to their citizens. One of the main results, published in a series of Overviews of Cochrane Reviews,^9-13 was that the evidence from randomised controlled trials (RCTs) was scarce in quantity and often low in quality. This surprising result showed that RCTs currently do not and in all likelihood cannot answer the range of research questions arising from practice information needs to provide all the evidence we need to treat our patients. There are two possible responses or conclusions from this observation. Firstly, more work may be required to substantially increase the number and quality of RCTs. Alternatively, RCTs alone maybe insufficient to develop an evidence based for rehabilitation and other study designs methods should be considered. Hence, one of the aims of the Meeting was to explore the utility of other research designs when RCTS are not feasible for whatever reason (such as ethical reasons, research question, topic, rehabilitation field, practice context or time circumstances).

The third lesson was learned during the COVID-19 pandemic and Cochrane Rehabilitation’s REH-COVER rapid living systematic reviews.^14-17 We conducted rapid reviews to synthesize the available information for clinicians who urgently needed to effectively treat patients with this new disease and its consequences, including the post COVID-19 condition. What we found is that a tremendous level of methodological confusion appeared toexist in rehabilitation research, with a mislabelling of study designs by most authors (i.e. using the wrong name to refer to the method used in a study). This observation clearly showed that rehabilitation research is mainly conducted by clinicians and that the research could be improved by engaging methodologists such as epidemiologists and biostatisticians to provide sound methodological guidance when designing and conducting clinical studies.

To summarise these experiences, the 5^th CRMM aimed to produce a helpful framework to guide rehabilitation researchers in the selection and use of appropriate study designs (beyond RCTs) for different research questions, clinical situations and in the reporting and publication of results. We aimed to develop a helpful tool for researchers and editors that could be implemented in the short term to enhance selection and reporting of designs and to inform funders and policymakers for their long term planning of research investments.

The 5^th Cochrane Rehabilitation Methodological Meeting papers

Following the classical organisation of the CRMMs previously described, seven international working groups prepared the first drafts of the papers, focusing on:

• the current knowledge on RCTs in rehabilitation coming from meta-epidemiological studies;

• observational study designs – specifically:

• the IDEAL (Idea, Development, Exploration, Assessment, Long-term study) Framework and its potential implementation in rehabilitation; and

• the Target Trial Emulation design;

• the Single Case Experimental Designs;

• the complex intervention studies;

• the health services research studies; and finally

• qualitative studies.

The six papers will be published in Special Sections of the European Journal of Physical and Rehabilitation Medicine in the current and the future issues. Moreover, two other papers discussed at the Meeting will complete the Special Sections:

• two experts on qualitative and quantitative research methodology collaborated in producing an original piece combining the two approaches with the perspective of:

• developing the right research question (answerable and logical); and

• identifying the proper study design for the research question; and

• the final summary paper on the “a road map to rehabilitation evidence production: the right study designs to identify the best available evidence.”

Specific teams prepared each paper, and all participants of the 5^th CRMM have thoroughly discussed them. They have been conceived as conceptual/discussion papers, even if some include original data. All teams had to answer the following questions:

• what study design should be used to answer clinically relevant research questions?

• How do we maximise the trustworthiness of the results to produce useful evidence on rehabilitation?

• How can we conduct studies of different designs to make them useful for rehabilitation end-users (clinicians, policymakers and patients)?

• When and how can these designs contribute to building the evidence in rehabilitation if RCTs are unavailable due to:

• the current research timeline (RCTs not available yet); or

• the topic (RCTs not possible on that specific topic for ethical or feasibility reasons)?

The final step was to initiate the design of a road map to rehabilitation evidence production to guide authors and editors. The aim was to produce a helpful framework to use the correct end-point study design (beyond RCTs) in different clinical situations and facilitate a quality publication. This paper was informed by on all the discussions that took place during the Meeting and will be the final paper of the series.