Forming a Research Question from a Multi-Center Database

Abstract:

It is not uncommon for individuals to ask biostatisiticians and epidemiologists to assist them with a research project. Often the request is in the shape of statistical analyses. However, most of these requests are nothing more than missed opportunities. This manuscript focuses on the reasons underlying such a statement. Most individuals might say that the most important aspect of a study is its conclusion. Many who would disagree with this sentiment and would feel that the most important aspect of a study rather is the question it intends to address. If this question is not articulated sufficiently, any additional information stemming from the study will most likely be irrelevant. Herein, some principles for formulating (successfully) a question from a multi-center database will be described.

“In my days as a skilled technician, I stood on the sidelines, directing the game and scoring. I was an observer and a manipulator of other people’s experience. Now I join in the game. I win and I lose; I live and I learn."—Jean McNiff, Action Research, Principles and Practice (1)

We have often heard of the “art of medicine.” Is the practice of medicine truly an art, or are there fundamental principles that guide the care of patients? Certainly knowledge concerning pathophysiology, biochemistry, anatomy, and other foundational learning is crucial to this formula. However, the application of this knowledge base into patient care is perhaps truly at the root of what is meant by the “art of medicine.” Similarly, the conduct of research is also an art. Craftsmanship and apprenticeship are inevitably involved through the application of foundational knowledge in the science of clinical research. Factors that come into play in this setting deal with the development of a research question, identification of an apt study design, and grounding the development of new science into the context of existing science.

The principles for formulating (successfully) a research question from an existing multi-center consortium, namely the Northern New England Cardiovascular Disease Study Group (NNECDSG; http://www.nnecdsg.org/) will be highlighted by providing a concrete example of how these principles have been implemented through this consortium. Nonetheless, as Jean McNiff had stated above, it is paramount that those that wish to truly succeed in this process roll up their sleeves and experience the process. Experiential knowledge will inevitably trump knowledge gained through reading words on a printed page.

NORTHERN NEW ENGLAND CARDIOVASCULAR DISEASE STUDY GROUP

The Northern New England Cardiovascular Disease Study Group (NNECDSG) is a voluntary research consortium composed of clinicians, research scientists, and hospital administrators. The goal of the group is to foster the continuous improvement in the quality of clinical care through the analysis of process and outcomes data and the timely feedback of information. The structure of the NNECDSG facilitates the development and implementation of new knowledge into clinical practice through our regional meetings that occur three times per year. At these venues, we share results from research and quality improvement projects and solicit feedback concerning the relevance and implications of our findings to date. The NNECDSG, governed by an executive body of representatives from each of our eight regional medical centers, is directed by Gerald T. O’Connor, PhD, DSc. A dedicated team of research analysts, administrative staff, and our project manager supports NNECDSG research projects.

TRANSLATION OF IDEAS TO RESEARCH QUESTIONS

Just as we were taught in elementary school, the key to science is reproducibility. One of the approaches that will consistently assist us in this regard is to begin the process by an individual developing a rough statement of intent. For instance, “I wonder what are the risks and/or benefits of a certain procedure or action.” Such statements often result from reading an article in the published literature and contemplating whether similar findings hold within our own regional experience. The process works more efficiently when these statements are put down on paper. Internally, our group convenes representatives from each of our eight institutions, plus our analysts and project manager, on a conference call to flesh out the idea. This process helps to turn the initial statement into a working research question. We additionally use this time to identify factors that might influence the association between the IABP and adverse sequelae.

Initially four items must be addressed.

1. What problem are we trying to solve?

2. What is the outcome that we are studying, and how do we define it?

3. What are the exposures that we think influence the outcome of interest?

4. What factors might confound the relationship between the exposure and outcome?¹

Seems an easy enough set of questions to answer, doesn’t it? Nonetheless, mastering this exercise is in part the “art of clinical research.” Our research team asks other probing questions to frame overarching questions, such as the following:

• What is the mechanism by which we suppose the exposure influences the occurrence of the adverse outcome?

• Can we identify disease and process factors that might affect our results?

• What answers can we provide from the data we already have at hand?

• What other questions might we need to ask to illuminate the association between exposures and outcomes?

Through these discussions, our research team develops a preliminary analytic approach for addressing the question at hand. Our team convenes a follow-up conference call to review the results of this analysis.

In the interim, our research team focuses its efforts as well on reviewing the peer-reviewed literature to determine what the state of the literature is related to the topic at hand. Our research team develops a series of queries to be executed against the National Library of Medicine (OVID; http://www.ovid.com/). An additional strategy to identify the breadth of evidence concerning a given topic is to identify articles that reference the literature found in your National Library of Medicine search (ISI Web of Knowledge; http://www.isiwebofknowledge.com/). Once these articles have been found, the NNECDSG shares the literature with the team members and empowers one of the members to review and summarize the literature for others.

On the follow-up conference call, our research analyst summarizes findings from the preliminary analyses. At this stage, it is pertinent and relevant that the clinical team members determine whether the analyses suitably address the research question at hand. The team must assess whether the analyses have face validity, especially if the analyses do not confirm, but perhaps dispute, previous findings in the literature. Generally speaking, the clinical team requests additional analyses, especially related to patient subgroups. The NNECDSG leverages our three regional meetings held every year to solicit critique, feedback, and general comments from our regional study group. In this way, our peers act as a sounding board to improve the overall research project before submission or presentation at scientific meetings or publication. The input gained through this process has been instrumental in refining our analyses, as well as developing the infrastructure necessary for translating our findings into changes in the practice of care. Often abstracts to scientific conferences are submitted after several iterations (assuming the topic and findings have sufficient merit), with the assumption being the development of a publication.

A BRIEF EXAMPLE

An intra-aortic balloon pump (IABP) is used to reduce myocardial work and oxygen consumption and improve coronary perfusion. Individuals within our region were interested in understanding the risks associated with inserting an IABP, especially in light of findings of a single-center randomized trial by Christenson et al. (2). Christenson et al. found a survival benefit among high-risk coronary artery bypass grafting (CABG) patients. Investigators from two institutions within Canada approached The NNECDSG to determine (i) whether a survival benefit existed within any patient subgroups and (ii) whether a multi-center database might validate the findings from the randomized trial of Christenson et al. Because of the cost of using an IABP, as well as the adverse sequelae associated with its implantation and use, there was sufficient interest to pursue investigating this issue.

The eight NNECDSG institutions along with two institutions within Canada began exploring this idea a little further. The principal outcome of this study would be in-hospital mortality. Preoperative insertion of an IABP was the primary exposure of interest. To validate findings from the trial of Christenson et al., analysts created seven high-risk subgroups. Last, propensity score analysis was created to assess the independent effect of IABP use on perioperative mortality (3). In short, propensity analysis seeks to create balancing across groups of patients in an observational study to mimic the effect of randomization. NNECDSG analysts performed the analyses with consultation from our Canadian colleagues. Along the way, both groups discussed the development and execution of the study vis-à-vis telephone conference calls and in person (at the NNECDSG regional meetings).

NNECDSG investigators (4) were unable to validate the findings from the randomized trial of Christenson et al. Indeed, use of a preoperative IABP was associated with increases in perioperative mortality. Although propensity scoring was able to simulate a randomized trial through balancing of covariates, use of an IABP remained a significant predictor of mortality.

Why should an observational cohort not reproduce the findings of a randomized trial? The group surely devoted an enormous amount of energy toward refining the research question. Nonetheless, we were unable to validate the findings reported by Christenson et al. Through this experience, we recognize that we were unable to capture fully the reasons underlying the insertion of a preoperative IABP. Some surgeons used the IABP as a rescue strategy, whereas others used the IABP prophylactically. Any conclusions that might be drawn about the potential beneficial effects of a pre-operative IABP are obscured by these differences in clinical decision making. Accordingly, although we were able to articulate adequately our research question, we were unable to leverage our registries to isolate subgroups of patients who would benefit, perhaps, from the use of an IABP. Since this time, we have added new fields to the NNECDSG regional registry to identify the reasons for the insertion of the IABP.

SOME ADDITIONAL POINTS

It would be remiss not to take some time to deal with issues of sample size and statistical power, because they are linked with the formulation of a research question. As a recipient of many research-related questions from well-intentioned individuals, the experience of having the unfortunate task of informing researchers that they have not recruited sufficient subjects into their study is not gratifying. The goal of research should be the discovery of new knowledge through a scientific process that affords others to replicate findings. Unfortunately investigators often consult with statisticians and epidemiologists to calculate sample sizes and statistical power after their studies are completed. These post hoc calculations are not intended to be informative to the process but rather satisfy an external agency’s curiosity. Halpern et al. (5) described two scenarios in which it is ethical to enroll subjects into an underpowered study: (i) when one expects to pool studies into a meta-analysis and (ii) early or exploratory studies of new drugs or devices. This author strongly encourages investigators to ground the development and execution of their research projects with calculations of sample size and statistical power. Otherwise, enormous amounts of energy will be expended without a meaningful or substantiated conclusion.

CONCLUSIONS

There are enormous benefits gained by patients through the development and throughput of research studies conducted by multi-center consortia. These groups afford themselves opportunities to identify variation in the conduct of care, and through quality improvement endeavors, rationalize the practice of medicine. The strength of these groups are not in their ability to conduct randomized trials but in the formulation and execution of sound observational studies.

It was the intention of this paper to describe some of the salient principles of formulating multi-center research studies. The points outlined in this manuscript do not represent the only path toward creating a sound research question. Nonetheless, this paper served to describe a generalizable model that has been developed and used by the NNECDSG. To be successful, however, research must be a team sport, with the active engagement of all parties involved in the care of patients. We have gained much through the multidisciplinary nature of our group, and the fruits of our work should certainly be attributed to the shared contribution of our nurses, perfusionists, surgeons, cardiologists, administrators, research personnel, and patients.

DISCLOSURE

The opinions expressed in this manuscript are those of the author and are not endorsed by the Northern New England Cardiovascular Disease Study Group.