Identifying Challenges and Opportunities in Clinical Research Informatics: Analysis of a Facilitated Discussion at the 2006 AMIA Annual Symposium

Abstract

Clinical Research Informatics (CRI) is a rapidly developing sub-domain of Biomedical Informatics that has seen considerable growth in recent years. While there are numerous activities and initiatives ongoing in this domain, systematic consideration and analysis of the challenges and opportunities that exist in this area are lacking. To begin to address this gap in knowledge and inform next steps in advancing this developing domain, we conducted a facilitated discussion among a diverse group of interested participants attending a meeting of the Clinical Research Informatics Working Group at the AMIA 2006 annual symposium. Findings from our analysis of these data are presented here and indicate a broad array of challenges and opportunities facing this developing area. These findings add new information to the limited literature regarding CRI and should provide direction for those working to set the CRI research and development agenda.

INTRODUCTION

Clinical research is critical to the advancement of medical science, and conducting such research is a complex, information intensive endeavor. Information technology solutions have long been applied to the work of clinical research, but interest in the interface between clinical research and biomedical informatics has grown considerably in recent years [1, 2]. As such, a new sub-domain of biomedical informatics, clinical research informatics (CRI), has emerged. Work in this domain is already proceeding at a rapid pace, and efforts like the Institutional Clinical and Translational Science Awards (CTSA), and the Inventory and Evaluation of Clinical Research Networks (IECRN) have significant informatics components that enable information sharing between groups working on CRI-related research and development [3]. Despite these and other efforts, there remains no consensus as to the grand challenges that face this new domain. Understanding these challenges is critical to setting the agenda and prioritizing work in this area.

Clinical Research Informatics

Clinical investigators and research staff face many challenges throughout the clinical research process, including designing protocol documents, recruiting subjects, implementing data collection instruments, and ensuring regulatory compliance [1, 4]. Numerous information technology (IT) systems addressing the needs of the clinical research environment have been developed. Such systems have been shown to have many benefits, including: 1) decreased time and effort required to develop protocol documents, 2) increased efficiency of subject recruitment, 3) increased data quality, and 4) decreased protocol deviations [1, 4]. However, the adoption rates of IT in the clinical research domain vary widely, which has been largely attributed to usability problems [5] and socio-technical challenges [6]. In order to better understand the relationships between the information needs of clinical researchers and available IT systems, it is helpful to conceptualize the conduct of clinical research as a multiple-stage model [1], consisting of: 1) hypothesis development, 2) protocol development, 3) participant recruitment, 4) data collection, 5) monitoring and quality assurance, 6) analysis and interpretation, and 7) dissemination and reuse of results. At each stage in this model, a combination of general-purpose, clinical and research-specific IT systems may be utilized. Examples of general-purpose and clinical systems that are able to support the conduct of clinical research include literature search tools, electronic health records (EHRs), statistical software and data mining tools, decision support systems, and computerized physician order entry systems. Similarly, research-specific IT systems that have been developed include tools for simulation and visualization, protocol authoring, participant screening and recruitment, and electronic data capture, as well as research-specific web portals. Underlying many of these systems are a multitude of conceptual, ontological and architectural models developed by standards organizations such as the BRIDG project, caBIG, CDISC, and HL7.

AMIA CRI Working Group

The Clinical Research Informatics Working Group (CRIWG, formerly the Clinical Trials Working Group or CTWG) is the home within AMIA for professionals who focus on the area of CRI. The group provides a forum for such professionals to network, collaborate, and collectively formulate CRI policy, research and development agendas within AMIA. Over the past 12 months, the leadership of the CRIWG has conducted a number of teleconferences and face-to-face meetings intended to assess member perceptions concerning the state of knowledge surrounding CRI research and practice. The overall impression from these discussions has been that our collective understanding of the grand challenges facing the rapidly developing domain of CRI is extremely limited. Therefore, we initiated the study described in this paper as a first step in a line of planned research to elucidate such challenges.

Objective

The aim of this study was to take a first step toward identifying the range of challenges and opportunities facing CRI in order to inform the direction and prioritization of research and development in this new domain, and to support the strategic planning objectives of the AMIA CRIWG.

METHODS

Given the early stage of knowledge in this domain and our stated objective, we employed qualitative research methods for this study. The specific methodology employed incorporates the conduct of a facilitated discussion, followed by the thematic coding and analysis of audio recordings, transcripts, and field notes generated during the course of a preceding discussion. This method was selected owing to its ability to identify and focus attention upon naturally emerging themes within a dialogue of subject matter experts, therefore mitigating potential investigator bias as much as possible while simultaneously increasing the quality of the resulting domain knowledge by incorporating and synthesizing multiple lines of reasoning [7].

Study Design

We conducted a single semi-structured facilitated group discussion, a qualitative research method during which a moderator guides a group of participants through a series of discussion questions in order to elicit their perceptions on a range of topics.

Participants

A convenience sample of interested AMIA annual symposium attendees participated in the facilitated discussion. While 46 people were in attendance and given the opportunity to contribute to the discussion, 22 actively participated. To encourage broad participation in advance of the session, participant recruitment involved posting invitations about the planned session on multiple AMIA mailing lists as well as posting flyers announcing the planned panel presentation and follow-up discussion around the meeting area of the annual session hotel.

Approval

The University of Cincinnati’s Social and Behavioral Institutional Review Board approved this study prior to its conduct. Given the low risk to the subjects, signed informed consent was not required, but an IRB approved information sheet describing the study was approved for distribution to study subjects prior to the initiation of the facilitated discussion session.

Setting

The study session was conducted during the second half of the 2006 AMIA annual symposium’s Clinical Trials Working Group business meeting. The meeting took place in a hotel meeting room on November 14, 2006. The facilitated discussion portion of the session was conducted during the final 30 minutes, following a series of presentations on various ongoing national and international CRI initiatives by expert panelists who then participated in the discussion along with audience attendees. Topics that were presented by these panelists included descriptions of the following initiatives: Global Trial Bank and WHO International clinical trial registry efforts, the BRIDG project, caBIG, CDISC, and the NIH CTSA effort. Presentation slide documents can be found on the AMIA CRI WG website <http://www.amia.org/mbrcenter/wg/cri/>.

Data Collection

Prior to starting the session, all prospective participants were notified of the planned study and provided with the IRB-approved information sheet mentioned above. Consent to have their comments recorded and analyzed in a confidential manner was implied if they remained and participated, though participation was not required for those who remained to listen. Further, to avoid any sense of pressure to participate, all were informed that they would have another opportunity to for participate after the session via the working group’s email list.

The facilitated discussion session was conducted by a researcher with experience conducting such group interviews (PE) using a semi-structured interview technique and the discussion was audio-taped. This approach was used to allow for the emergence of unanticipated relevant issues while still ensuring coverage of topics of anticipated importance based upon the preliminary discussions within our working group. The discussion session began with a brief reminder of the research nature of the discussion and a mention of the purpose of the discussion. This was followed by open-ended questioning of the group about their views on three major issues: 1) the challenges facing CRI; 2) the opportunities facing CRI; and 3) the role that AMIA could play in this domain. Open-ended follow-up questions were asked as needed to delve more deeply into particular issues. Only when these techniques yielded little or no response were closed-ended questions used to further explore issues of potential importance as anticipated by the investigators prior to the session. Microphones were provided to all participants in order to capture each word clearly. The audio-taped recording was professionally transcribed verbatim and the audio file was converted to an MP3 in order to facilitate audio-review during the analysis phase by the investigators as needed for verification of the transcript. Field notes were manually recorded during the session by an experienced investigator (PP).

Data Analysis

Three pages of field notes and seven single-spaced pages of verbatim transcript of the approximately 25-minute recording were analyzed by two reviewers (PE, PP). A sentence or phrase in the transcripts served as a unit of analysis for coding purposes. Content analysis was performed using a grounded theory approach to identify emergent themes. Descriptions for each unit of analysis were based upon participants’ utterances whenever possible. After each researcher conducted his independent review of the transcripts and notes, the investigators entered an iterative process beginning with discussion of their descriptions of the data. This led to agreement on broad categorizations that could be directly linked to the raw data. As a final step, the categorizations were organized into common themes and summary interpretations were drawn from the findings.

RESULTS

Twenty-two meeting attendees participated actively in the facilitated group discussion. In order to maintain the confidentiality of this relatively small group of participants, detailed demographics cannot be presented. We can report that participants represented a broad range of backgrounds and perspectives including: academic informaticians, clinical and translational investigators, governmental funding agency representatives, leaders of major health and research informatics initiatives and professional organizations, pharmaceutical and health IT industry professionals, and health IT investors. All participants were directly involved in biomedical informatics-related and/or clinical research-related activities as a major focus of their jobs. Participants responded spontaneously to the open-ended questions posed, with little need for the interviewer to prompt responses via the use of follow-up questions. As such, there was minimal need for closed-ended follow-up to discussion points.

Emergent Themes

Analysis of the data yielded numerous distinct categories of comments related to challenges, concerns and opportunities facing CRI. From these, twelve themes emerged that together encompass all comments made during the discussion. While some data elements could be assigned to unique categories, many represented more than one concept and were therefore cross-categorized. For example, if a comment addressed the need for education about data standards, it was coded as relating to both education and standards. The main findings, including these 12 broad themes, their respective underlying categories and representative quotations for each theme (but not each category due to space limitations), are organized and presented in Table 1.

Table 1.

Results - Emergent Themes, Their Underlying Categories, and Representative Quotations

Themes	Underlying Categories	Representative Quotations
Research planning and conduct	Improved research planning tools Advanced clinical research design Tools cumbersome/poorly integrated	“...the issue is that the tools are cumbersome and poorly integrated with the workflow of patient care, and so you can’t recruit either the investigators or the patients.” “(Investigators) over-promise because the work is too hard to accomplish with the tools they are given”
Data access and integration	Poor data access Lack of data integration Secondary use of data issues	“I think one of the largest challenges institutionally is data access and integration, and that’s the barrier for the investigator.” “…the interface between the electronic medical record and the clinical research is critical.”
Educational needs	Educate students, investigators, clinicians about CRI issues Educate those working in CRI Educate informaticians about CRI	“…it might be worth considering what is the training required, because it is a tremendous leap for a DBA to understand the regulations for running a clinical trial. It is a tremendous leap for an electronic health records guy. CIOs and academic centers do not get it. No offense. This is a big leap here.” “...(We should) educate some of our informatics colleagues… about the obstacles and roles and needs that we have in this research environment.”
Fiscal and administrative issues	Research billing challenges Costs of research software Improved business processes Lack of incentives for adopting research tool	“We are struggling with the appropriateness of tracking research costs and research charges and making sure that we are compliant with that and what is research and what is not. So, there is a whole other set of business processes on the investigator’s side that need to be somehow linked and coordinated with processes on the study side” “So one of the big things driving this is the complete lack of money or incentive to move beyond paper.”
Investigator participation	Investigator recruitment challenges	“So the issue again rests on what can we do to make it easier so that clinicians are interested in becoming investigators.”
Leadership and coordination	Need for coordinated CRI agenda Need for setting practical goals Need for coordination between initiatives/among stakeholders Desire for leadership/guidance from professional organization	“…(the part) AMIA can play a role in is keeping up with all of this… so creating this portal that allows people from this group to be able to contribute and to go and understand what is going on in this space.” “(We need to encourage) open comment from all the different perspectives so we are hearing from the investigator, the vendor, the institution, the NCI and other NIH institutes; it would be fabulous to get that input (about CRI initiatives).”
Subject recruitment	Ineffective subject recruitment Current tools make recruitment difficult Lost opportunities to recruit	“Recruitment is the single biggest challenge. The investigators always over-promise. They do feasibility assessment. They try to target as best they can but there is under-performance that causes them to have to do rescues mid-way through. They have to close down sites, start up new sites. They cannot get a good handle on how to predict or estimate, or enhance and augment recruitment.”
Scope of CRI	Extend field beyond clinical trials Include support for nursing research Include research partners in agenda (e.g. pharma, government,)	“(We should) expand our vision from clinical trials to encompass all of clinical research, because I think this is going to be critical as we move forward. It is not just the trials. It is the outcomes, the biomarkers. It is the epidemiology studies.” “Nurses use a wide range of research techniques, and I think we need to be investigating those as well.”
Socio-technical issues	Research/clinical missions not integrated Inadequate resource allocation Poor stakeholder collaboration	“There are some obvious technical challenges (to providing data integration and access), but… one of the biggest challenges coming is actually integrating the research mission into the academic health care environment and ensuring that (removal of) these barriers that we have to data and systems and use of them for research is able to happen.”
Standards	Need for CRI data standards, models Apply clinical standards to research	“...properly representing medical concepts and the right terminology in order to access anything down stream which includes real world data from insurance and electronic medical records, databases, as well as other knowledge bases.”
Workflow	Integrate tools into workflow Inefficiency of research processes, need for effective models IRB/regulatory challenges	“...we all complain about the time it takes for IRB review.” “(Current systems suffer from) a complete lack of integration between patient care processes and clinical research.” “While a lot of talk has been made about standards, (so) we can move data back and forth, what is actually happening is there’s no work processes that make that happen.”
Lessons not learned	Concern about repeating clinical informatics mistakes Need for best practices Under-use of current tools/best practices	“…the problems and the hopeful solutions (being proposed) are so analogous to the issues that have swirled around the electronic health record here at AMIA for the last 15 or 20 years or so, and I would be a little more comfortable if more was said about what went wrong with that. Because tostart all over again with the same good hopes and the same stuff about standards and we will make software and all the rest of it, it didn’t cut it the last time, so I am hoping that the lessens from the past will be incorporated into the work on the clinical research (agenda).”

DISCUSSION

Clinical research informatics has emerged as the biomedical informatics sub-domain focused on addressing the myriad informatics issues facing today’s complex clinical and translational research environment. As might be expected given the early stage of work in CRI, numerous challenges exist and were identified by participants during our facilitated group discussion. Interestingly, while several of these challenges have been the subject of considerable discussion in the field and were anticipated by the investigators (e.g. issues of data access and integration; need for improved research tools; inadequacy of current subject recruitment efforts; importance of standards), several other important challenges that emerged from our analysis have received relatively little attention to date in the published literature or other common venues. One such example was the repeated call for enhanced leadership and coordination among informatics efforts within CRI domain, as well as between CRI efforts and other more general biomedical informatics initiatives. Another example related to the numerous comments concerning the need to educate various stakeholders including investigators, researcher staff, IT staff and other informaticians about CRI issues and best practices. Several categories of comments also reflected the need to ensure that CRI’s scope was broadened to encompass the entirety of clinical and translational research, in order to ensure that the full benefits to medical science are realized. Also, beyond the need for technological solutions, there was a clear demand to address the socio-organizational issues that often impede the degree of integration and support needed for CRI efforts to succeed.

Another finding that appears critical to the future success of CRI and was emphasized by participants related to the insight that many of the current issues in the CRI domain are analogous (if not identical) to those that arose in the past several decades with regard to clinical information systems in general and EHRs in particular, and that we must learn from these past lessons.

Perhaps the most important finding derived from our analyses is the common impression that, despite the presence of a national agenda for clinical and translational research, including a growing call for correlative work in CRI, a coordinated research agenda focusing specifically on CRI is conspicuously lacking. As a result, the concerns raised speak to the fact that most CRI research and development has tended to be narrowly focused (from either an application or disease-specific context). Several of the emergent themes indicated that, while some such focus is certainly needed, it must not be achieved to the exclusion of also collectively developing the higher-level, extensible informatics frameworks and infrastructure needed to address the requirements of the clinical and translational research domain.

Limitations

This study has some limitations which should be noted. First, we collected our data during a single session, so it is not possible to know if further sessions would have led to the identification of additional themes/categories. Further, we studied a convenience sample of subjects. Although these subjects appeared to represent a broad group of stakeholders, there may have been perspectives not represented. These limitations will be addressed in part by our next steps as outlined below. It is also possible that the initial panelist presentations provided prior to the discussion may have influenced the participant comments in an unexpected manner. Finally, due to the nature of the methods employed and despite attempts to avoid bias (e.g. dual independent analysis), some bias or subjectivity may have influenced our analysis or results.

Next Steps

This study represents the first step of a more comprehensive research plan to identify and illuminate the range of challenges and opportunities facing CRI. Additional steps include: 1) analysis of data from follow-up discussions on this topic on the CRIWG email list; 2) performance of post analysis data verification using complimentary qualitative and quantitative methods; and 3) surveying the CRI community to determine the degree to which stakeholders share the perceptions identified. Finally, prompted in part by these findings, we intend to continue working within AMIA and the CRIWG to develop a plan for the kind of CRI coordination and support called for by these findings.

CONCLUSIONS

Clinical Research Informatics has emerged as a distinct sub-domain of Biomedical Informatics, and there are numerous challenges facing the field. There are multiple ongoing activities in this domain at the national and international levels that are addressing several of the important challenges and opportunities identified in this study. However, this study indicates that there are other issues of importance to ensure the successful progress in CRI that are not currently the subject of great attention and have not previously been reported in this context. These findings should inform the development of an agenda for this emerging domain. Further work is ongoing to verify these early findings and assure that the full range of challenges and opportunities in this emerging domain are identified.