Measuring Impact: Citation Rates and the Quest to Identify Meaningful Clinical Research in Critical Care Medicine

“If you think you’re too small to have an impact, try going to bed with a mosquito.”

Annita Roddick

In times of increasing costs and limited funding for clinical trials, identifying trial designs that generate meaningful data to improve the care of patients in the intensive care units is crucial. In this issue of Critical Care Medicine, Marshall and colleagues present an analysis of 391 clinical trials each enrolling more than 100 critically ill patients, that seeks to identify which design factors (such as source of funding, type of study, and the research model) affect the overall impact of a given study, as measured by how often it is cited in the literature.¹ Studies conducted by investigator-led clinical trials groups or consortia, as well as those addressing a comparative effectiveness research question, were independently associated with significantly higher annual citation rates. In contrast, the funding model (industry versus peer review funding) and the specific study focus did not correlate with higher citation rates. The investigators’ efforts are ambitious and should be applauded; their analysis provides interesting insight into the relationship between specific trial characteristics and the likelihood of producing widespread discourse within the scientific community. At the same time, this study raises questions about the meaning of citation rates and whether this metric is an adequate tool for identifying clinical research that has the greatest positive impact on patient care.

While citation rates have long been used to judge the efficiency of grant funding and as criterion for academic advancement, there are limitations to using this approach. Notably, citation rates often favor older studies compared with more recent trials (a lead time advantage). Furthermore, one needs to look below the surface to understand why a study has a high citation rate. For example, examining the 16 most cited papers identified by Marshall et al.—9 of which were led by a trials group and cited more than 100 times annually—reveals the potential pitfall of assuming that studies with high citations rates have had a positive and lasting impact on improving patient care. Four of the sixteen studies that showed a benefit of either tight glucose control,^2,3 steroids and the utility of ACTH stimulation testing,⁴ or activated protein C⁵ were later refuted by four other studies on the same list.^6–9 Furthermore, the trial of early goal-directed therapy—the study with the second most median citations per year—has recently been refuted by three large multicenter, clinical trials.^10–12 Ironically, in most cases (3/4) the paper that disproved the original trial results and limited the use of potentially harmful interventions—arguably more meaningful and durable work—was cited less often. As well, some highly cited studies have design flaws that make the interpretation of the results challenging and thus difficult to apply at the bedside. For example, the study designs for both the ARDSnet trial¹³ and the TRICC trial¹⁴ have been questioned for discounting the titrated nature of standard practice.¹⁵ The lack of such a usual titrated care control arm limited the ability to assess the clinical impact in the ARDSnet and TRICC trials of the two treatments tested.^16,17 In total, these findings would suggest that citation rate does not necessarily identify clinical trials that advance and improve patient care. In some cases, high citation rates may even be indicative of controversial, non-reproducible science that is “too good to be true.”

Since the creation of The Canadian Critical Care Trials Group in 1989,¹⁸ a number of other investigator-led consortia have formed and are dedicated to designing and conducting clinical trials in the field of critical care.^19,20 These groups have led numerous multicenter, randomized controlled trials in the past two decades, generating valuable clinical research. The role that these groups have had in advancing the understanding of critical care medicine therapies is unquestioned and their willingness to design and conduct clinical trials in critically ill patients, a population that has often been neglected by industry, is commendable. However, trial groups are only one source of valuable data: multiple streams of clinical research are most likely to advance knowledge and improve patient care. While no one would argue that smaller trials are inherently superior to larger trials,²¹ it is important to recognize the ingenuity and the contribution of studies that were designed and carried out by investigators not involved in such groups, sometimes performed at single centers. Examples of small studies that have had a lasting impact on patient care include the first studies showing the benefit of bedside ultrasound for critical care procedures,²² the use of steroids to treat severe Pneumocystis jiroveci pneumonia in AIDS patients,²³ or the use of transluminal balloon dilatation for coronary artery stenosis.²⁴

Unfortunately there is no perfect instrument for identifying quality, high impact clinical research and it is important that we recognize the inherent limitations of citation rates and the ongoing need for peer review. Trial groups, small investigator-led teams and industry bring talent, insight and perspective to many of the clinical problems faced by intensivists and the patients they care for. Whether their work proves meaningful to patient care depends mostly on the relevancy of the question, the quality of the trial design and the reproducibility of the findings.