The increasing trend of observational studies in clinical research: Have we forgotten and started defying the hierarchy?

In the contemporary era of evidence-based medicine (EBM), the question arises as to what would constitute the best research evidence. Editors of high-quality journals have a variety of manuscripts flung at them, and for them, selecting a good manuscript is like identifying ‘a cygnet from a paddling of ducklings’. Study design, which is often not correctly formulated, plays a dominant role not only in interpretation but also in the acceptance and publication of research. It has been mentioned time and again, that faulty study design can be a cause for manuscript rejection.[1] There exists a general hierarchy that ranks the type of research based on the validity of the findings and effectiveness of the intervention. There are several types of study designs, and randomised controlled trials (RCTs) were initially ranked as the highest level of evidence which has been currently replaced by meta-analysis and systematic reviews.[2] [Figure 1] In another recent modification, Murad et al. removed them from the pyramid and suggested them to be a lens through which evidence is reviewed.[3]

Figure 1.

Hierarchy of Level of Evidence (RCT:Randomised controlled trial; SR: Systematic review; MA: Meta-analysis)

WHY ARE RCTS RATED SO HIGH?

Clinical trials generally aim to ascertain whether any intervention, program, or drug treatment brings about a positive outcome for the recipients. An RCT is a prospective, comparative, interventional study of quantitative nature performed in controlled conditions in randomly selected subjects.[4] It provides the most dependable evidence of the efficacy of preventive, diagnostic, and therapeutic interventions because of the robust and rigorous scientific techniques used to ascertain a cause-effect relationship between intervention and outcome.[2] RCTs are ubiquitously regarded as the ‘gold standard’ of biomedical research. However, when the population being studied for research is divided into groups, there is a chanceof imbalance between them with respect to known or unknown confounding factors even before a comparison is started. Randomness and blinding in the selection process are important aspects to eliminate biases, ensure the veracity of the results in clinical research and maintenance of scientific reliability.[5] The utility of a control group makes these randomised trials the most powerful research tools in the current scientific era. A ‘double-blind’ study, in which the concealment of allocation is maintained till analysis, removes our all-too-human misery, optimism, and aspiration from the comparison. However, these aspects have often been found to be deficient in the studies submitted for peer review; nevertheless, several workshops and sessions on ‘Research Methodology and Publication’ are being conducted in various national and state conferences and Continuing Medical Education (CME) activities which provide a basic understanding of biomedical research and statistics.

DEFICIENCIES OF RCTS

A rose, however, beautiful has thorns. Similarly, though RCTs constitute the best quality evidence, it is possible only when their methodology is rigorous, and they are conducted in a scientific manner. Hence, not all RCTs qualify as a high level of evidence and while grading them, the presence of selection bias, inadequate sample size, improper randomisation, lack of blinding, inconsistency, imprecision, not performing intention to treat analysis or not following Consolidated Standards of Reporting Trials (CONSORT) statement can lead to a serious deterioration in their quality and downgrading of the evidence generated by them.

A meticulously designed clinical trial is not always practical, and it has been documented in the past that the quality of the randomised trial and randomisation process itself is not always properly maintained. There is ample evidence that the quality of randomisation is often suboptimal in RCTs, and these are not truly randomised clinical trials.[6]

It may not be practical to perform RCTs in some scenarios such as trials where results are dependent on patient cooperation or conditions for which there is already an established gold standard for treatment. It may be difficult to attain and conceal a true placebo in an RCT. In fact, it may be unethical to give a placebo in certain situations such as in a trial on a life-limiting illness. A placebo may not be justified if it would mean refuting the participant's usual management.

Further, animal research can be inadequate for making recommendations for or against the treatment in humans as the response may not be identical in many circumstances. In fact, authorisation of medications based on animal research has led to substantial harm to the human population in the past. The animal studies may be conducted in conditions that would never be used while conducting anaesthesia on human infants and children. Also, biologically, postnatal growth and development in animals and human infants do not match even in similar age-groups. Huge costs are often incurred in conducting RCTs. Rarely, the results come out to be so beneficial that it would be considered unethical to deprive the control group of its benefits. In some medication trials, the investigators must terminate the trial prematurely because of harmful or negative results in the interim analysis. Sometimes, the results may not mimic real-life treatment, and the subjects will not reciprocate the highly controlled environment involved in the selection and exclusion of patients

As it is not always feasible to conduct an RCT, researchers have to, at times, resort to other trial designs. It has also been contended that it is not always possible to guide every clinical decision on RCTs.[7]

WHAT ARE THE ALTERNATIVES TO RCTS?

There are other forms of interventional studies which are not categorised as RCTs but are ranked higher than their observational counterparts.[8] When RCTs cannot be used, the interrupted time series design (ITS) is a possible alternative. Many a times, intervention can be done, and the pre- and post-effects can be compared without a control group. This issue of the Indian Journal of Anaesthesia (IJA) has one such study wherein the pre- and post-effects of an intervention have been compared.[9] The main objective of ITS is to examine whether the data pattern observed post-intervention is different from that of pre-intervention.[10] Sometimes, a control group can be an active group receiving another form of treatment instead of a placebo to avoid ethical problems and to provide additional benefit for the participants in the control group. Few studies need not require a control group to validate the results. Such studies follow a single group interventional design as has been followed in the study on paediatric thoracotomy cases published in this issue.[11]

ROLE OF SYSTEMATIC REVIEWS AND META-ANALYSES

Many of the scientific articles that are published have contradictory results, lack certainty, and many results have been refuted over time. In such circumstances, clinical decision-making can be guided only by the reconciliation of studies and reviewing them scientifically. A systematic review uses a transparent process to identify and compile explicit results of multiple studies for a specific focused question after critical appraisal of these studies. These data can be pooled in a graphic form usually as a forest plot also known as the blobbogram, where the point estimate and 95% confidence intervals are presented on an individual line.[12] An interesting observation is that the number of systematic reviews has increased drastically over time with initiatives such as the Cochrane collaboration providing assistance and free software which helps in the process of compiling the data and providing meaningful results. Bias unique to this type of research is publication bias. The chance of publication is more for studies with positive results and in the English language. This forms the major difference between systematic reviews and narrative reviews. A well-conducted systematic review attempts to reduce such bias.

Whenever there is a difficulty in doing multicentre trials which include a large number of patients, a meta-analysis can be conducted. The word ‘ meta’ stands for ‘beyond’ in Greek. This tool, which has a systematic approach, analyses the results in a transparent manner by combining many studies, thus, increasing the sample size. It helps us to draw meaningful conclusions, resolve conflicting results and clear uncertainties. Meta-analyses provide results with increased precision and true estimation of the effect size.[13] As has been mentioned time and again, in the IJA, systematic reviews and meta-analyses get top priority in the publication process.

SO WHERE DO OBSERVATIONAL STUDIES OR SURVEYS STAND?

The great agnostic Robert G Ingersoll had stated ‘reason, observation and experience’as the holy trinity of science. One has to always observe to identify and analyse which is the first step in the practice of any research. It is the process where a question leads to many more questions and many more possibilities. Such observations can imply associations and lead to concepts. One has to evaluate to establish such concepts and intervene to solve the problem. Understanding the natural history of the disease and its complications is an area of research exclusive to observational studies. The Mayo Anesthesia Safety in Kids (MASK) and Pediatric Anesthesia Neurodevelopment Assessment (PANDA) are some of the historical observational studies which have helped to provide additional information which cannot always be evaluated using a randomised study design.[14,15] Though the General Anesthesia or Awake-regional Anesthesia in Infancy (GAS) study was performed as a muliticentre randomised controlled equivalence trial assessing the effect of an hour of sevoflurane anaesthesia exposure and suggested that it is safe in infants, the effect of multiple exposures could only be determined by observational studies due to ethical constraints.[16] Observational studies (cohort, cross-sectional studies or case-control studies) have the potential for a higher external validity than RCTs which means that their results can be extrapolated to a broad population to a higher degree.[17] However, they have a lesser internal validity because of the possibility of bias and confounding variables.

Nonetheless, the limitations in observational studies can be overcome to a certain extent by few statistical methods. Identification of potential confounders can be done using logistic regression analysis, and the effect of these potential confounders can be offset by propensity analysis scoring and, thus, reducing the selection bias.[18]

Feinstein and Horwitz had in their article stressed that RCTs may be an ‘elusive ideal’ and stated that the high-quality prospective cohort or retrospective epidemiologic studies offer a sustainable alternative for the (many-a-times) unachievable scientific gold standard of an RCT.[7] They emphasised the need to conduct other clinical epidemiological research designs in a very rigorous manner.

Another alternative is to set up international registries in which the registration of various data would be necessary. Though not free of biases, they have various advantages viz., no need for informed consent, data collected in an efficient manner, the feasibility of including large sample sizes, less costly or time-consuming, rare diseases or manifestations can be studied, and easier analysis of long-term effects.[19]

CASE SERIES AND CASE REPORTS: WHAT IS THEIR WORTH?

The least ranked form of research, that is, case series and reports are descriptive studies that describe clinical circumstances of individual patients and focus on the contextual analysis of events or conditions and their relationships. They are often not given preference by major reputed journals. Recently, numerous open access journals which exclusively publish anecdotal cases and series have come up.[20] These rekindle the interest and evoke curiosity among the readers. However, the utility of such knowledge in routine clinical practice remains questionable. In a recent amendment, the National Medical Commission has added ‘cases series’ in addition to the pre-existing criteria of ‘original research articles’ including systematic reviews and meta-analyses for academic promotions in medical colleges. From a reader's point of view, a letter to the editor is a much more popular, effective, precise and authentic means of conveying the relevant message as compared to a case report and that too in minimal words.

QUESTIONNAIRES AND SURVEYS: CURRENTLY POPULAR RESEARCH TRENDS?

Questionnaires and surveys are inexpensive and quick research tools applicable for the assessment of the current knowledge among clinicians as well as an assessment of the impact and reach of certain treatment methodologies or guidelines. Surveys also help to identify deficiencies and focus on areas of improvement.[21] A survey study in this issue of the IJA focuses on the improvement in postgraduate medical education in the country.[22] Electronic surveys are common and are easy to conduct. The coronavirus disease (COVID)-19 pandemic saw a multitude of e-surveys conducted conveniently during the lockdown period. Two more surveys published in this issue: a survey describing the practice of oropharyngeal pack insertion and removal practices amongst the anaesthesiologists of our country and a survey by showcasing extubation practices among anaesthesiologists are an attempt to improve safety and quality in health care.[23,24]

Questionnaires have their own set of limitations and biases such as self-reporting bias and social desirability bias. There is an article in this issue of the IJA that deals with the development and validation of a questionnaire.[25] This issue also has two interesting studies related to assessment and learning in postgraduate medical education, and they have questionnaires with a variety of questions in their study methodology for the study participants.[26,27]

REGISTRATION OF RESEARCH STUDIES: A VOLLEY OF DOUBTS?

Registration of clinical trials is an important process to ensure transparency and credibility of the study and, thereby, improving the quality of evidence. Awareness about the registration of clinical studies is minimal among young researchers, and the purpose of registration is not completely understood by many. There are certain misconceptions involving the registration of clinical research. The IJA has been flooded with manuscripts in the past year with authors pouring in queries and doubts regarding the registration of their studies. Do observational studies, retrospective studies and surveys need to be registered? Can Clinical trial registry of India (CTRI) non-registration be a reason for manuscript rejection? CTRI is a domain thatprovides registration of clinical research performed in the country. This was launched in the year 2007, and the registration process which was initially voluntary has been made mandatory since 2009. Subsequently, editors of major journals in India, including the IJA have declared that only registered trials would be considered for publication. Any trial involving human participants, of any intervention such as drugs, devices, surgical procedures, preventive measures, lifestyle modifications, educational or behavioral treatment, and rehabilitation strategies is expected to be registered in the CTRI before enrolment of the first participant. The registration process is easy if an ethics committee approval is obtained. The committee approving the ethical perspectives of the study should be registered with the Drug Controller General of India (DCGI). Registration of certain phase 1 trials requires direct approval from the DCGI. At this point, it is important for all researchers (past, present and future) to note that registration is not limited to interventional clinical trials but also extends to observational clinical studies, post-marketing surveillance trials, bioavailability and bio- equivalence studies. International multicentre trials can be registered with international registries like the Clinical Trials Register hosted by the United States National Library of Medicine. These studies should also be registered additionally in the National Register. (ctri.nic.in) [28]

WHAT NEXT? THE KIND OF RESEARCH TO BE TAKEN UP…. (RCTS?OBSERVATIONAL STUDIES? SURVEYS?)

The year 2020 stands as evidence of various complexities and adversaries of clinical research. The withdrawal of published articles on hydroxychloroquine from The Lancet stands proof of bias, errors and lack of transparency in research. We were guided throughout the course of the coronavirus pandemic by the RCTs pouring in from all over the world. In the race of evidence, the supremacy of the quintessential RCTs over cross-sectional surveys and observational studies is undeniable, but the scientific evidence has its own value depending upon the research question and the methodological rigor. In 1998, René Favaloro, a revered cardiac surgeon, while signaling on the long debate about the need for RCTs for bypass grafting, had aptly mentioned, “RCTs have developed such high scientific stature and acceptance that they are accorded almost religious sanctification. If relied on exclusively, they may be dangerous.”[29] This sets us thinking. Is it time for us to adopt and accept a diversity of approaches in research study and design?

To conclude, one thing is certain. RCTs or meta-analyses……Surveys or observational studies…the clinical research journey has to continue and move on.