Achieving palliative care research efficiency through defining and benchmarking performance metrics

Abstract

Purpose of Review

Research efficiency is gaining increasing attention in the research enterprise, including palliative care research. The importance of generating meaningful findings and translating these scientific advances to improved patient care creates urgency in the field to address well-documented system inefficiencies. The Palliative Care Research Cooperative Group (PCRC) provides useful examples for ensuring research efficiency in palliative care.

Recent Findings

Literature on maximizing research efficiency focuses on the importance of clearly delineated process maps, working instructions, and standard operating procedures (SOPs) in creating synchronicity in expectations across research sites. Examples from the PCRC support these objectives and suggest that early creation and employment of performance metrics aligned with these processes are essential to generate clear expectations and identify benchmarks. These benchmarks are critical in effective monitoring and ultimately the generation of high quality findings that are translatable to clinical populations. Prioritization of measurable goals and tasks to ensure that activities align with programmatic aims is critical.

Summary

Examples from the PCRC affirm and expand the existing literature on research efficiency, providing a palliative care focus. Operating procedures, performance metrics, prioritization, and monitoring for success should all be informed by and inform the process map to achieve maximum research efficiency.

Introduction

The Palliative Care Research Cooperative Group (PCRC) was founded with a commitment to clinical research deeply rooted in efficiency and the imperative to quickly translate meaningful research results to the clinical frontline.¹ Using the Institute of Medicine's (IOM) vision for effective and efficient future cooperative groups, collaboratives, and networks, we designed the PCRC to be highly responsive and nimble, conduct cutting edge scientific studies using rigorous methodology, and quickly disseminate evidence-based results to improve palliative care. To be efficient, we reviewed extant literature, examined the PCRC research portfolio and prior experiences of our investigators, developed responsive internal processes, and planned for iterative improvements to our approach.

Planning for Research Efficiency

Research efficiency is defined as the process to achieve maximal generation of meaningful evidence given a specific investment in study design, conduct, analysis, interpretation and dissemination. When core research tasks get bogged down in inefficient processes, resources (personnel time and money) are squandered, delays to the project timeline are more common, and the time to disseminate findings, particularly evidence to improve care is prolonged. The federally-funded cancer clinical trials system in the United States, which has been under significant recent scrutiny, is an often-cited example of the risks of inefficient systems. According to the IOM², the US Cancer Cooperative Groups are nearing a point of crisis that will only be avoided if significant improvements are made in research efficiency and effectiveness. The IOM noted that the cooperative group infrastructure operates sub-optimally due to redundant processes that delay the conduct, and, ultimately, translation of clinical research to improved patient care. Examples include an average of 920 days to progress from concept approval to activation of a Phase III Cooperative Group trial, with less than half of the activated protocols achieving 25% of their target accrual goal.³ The fact that only about 60% of trials funded by the National Cancer Institute are successfully completed and published² is a stark indicator of the need for improvements in research efficiency to realize the desired return on investment. Meanwhile, higher education and community health organizations may be hamstrung by bureaucratic inefficiencies, opening the door to private clinical research organizations that claim short timeframes from concept to “first patient / first visit.”

To achieve research efficiency, we must first understand the needed processes, ensure they are optimized and streamlined at the onset, monitor progress by developing relevant metrics and benchmarks, and iteratively enhance processes identified as inefficient. The goal is to spend the minimal amount of time, energy, and resources, yet yield scientific advances that result in significant improvement in clinical care. Increasing the number of trials completed on time and within budget is one expected outcome of these efficiencies, as is the preparation and distribution of findings in a timely manner. Likewise, we need to learn from established cooperative trial groups and adopt and/or improve upon their accomplishments. If efficiency in palliative care studies is not optimized, the available resources will prove insufficient and the pace of advances in palliative care will decelerate.

Using current literature and PCRC case examples, we describe approaches to maximizing palliative care research efficiency and thereby positively impacting patient care. Topics include process mapping, operating procedures, performance metrics, and monitoring for success. At the most basic level, optimal research efficiency is the point at which the investment to return ratio is fully maximized.

Process Mapping

Various approaches have been taken to map the processes required for clinical trials initiation and conduct, and then the translation of research into practice. A number of concept maps have been proffered to evaluate the overall translational research enterprise, such as the translational model from Trochim and colleagues.⁴ Here we focus more narrowly on the process mapping of functions in a clinical trials cooperative group. According to the review of cancer clinical trials by Dilts and colleagues, clinical trials conducted within cooperative groups face three types of administrative barriers: procedural, structural, and infrastructural.⁵ Procedural barriers include the many processes or series of steps required to open a clinical trial (e.g. protocol review, forms development, etc). Structural barriers are those that are related to organizational design, frequently caused when principal bodies involved in the study follow dissimilar procedural steps. Infrastructural barriers are those that occur due to the design of the core system's underlying foundation and interconnections. Dilts et al. defined each of these barriers in the form of a process map (Figure 1); a heuristic tool showing all steps involved in a process, making determination of which components are valuable (those that impact study quality, safety, or effectiveness for example) and which are non-valuable transparent.⁴

Figure 1.

Level 0 process flow map for opening an oncology clinical trial. (As a result of printing restrictions, Figure 1 is a highly aggregated view of the process flow of opening a clinical trial. For a more comprehensive view, go to http://www.cmrhc.org/ClinicalTrialsProcess/ProcessMap.pdf.

PI, principal investigator; CTO, clinical trials office; IRB, institutional review board; SRC, scientific review committee; CRC, clinical research center; FDA, Food and Drug Administration; Y, yes; N, no.

Depicting all process steps required to initiate, conduct, analyze and report on a trial in a single diagram provides a comprehensive view of the entire process and facilitates identification of unnecessary and/or redundant components, informing rapid and dramatic process improvement. Process mapping identifies functions that might be conducted simultaneously, in parallel, or consolidated into fewer steps. Exposure of the iterative steps and redundant feedback loops informs the design of relevant performance metrics. In essence, each step toward the initiation and successful completion of a clinical trial is an outcome and, therefore, a metric of its own.

The most time-intensive barrier to study implementation is synchronicity, the “compilation of various components of a concept or a protocol before it can be submitted to other participants in the process.”^{6,pg 3429} Examples of these components include the creation, review and finalization of the protocol, case report forms (CRFs) , operating procedures, and data management, including defining common data elements. Each of these tasks may be led by a subset of the research team, which then must communicate with a larger group and those with the ultimate authority to approve or revise. This review cycle typically involves a great deal of time-intensive exchanges requiring a point person to shepherd the reviews and monitor changes. In addition, many operating procedures are related to and dependent upon each other, obliging the review team to keep these inter-connections in mind and consistently evaluate the steps within and across operating procedures. Failure to synchronize these processes may result in delays in study initiation and resource-intensive mid-study changes and refinements, impeding the study's progress with additional feedback loops and review cycles.

Process mapping in action - Palliative care research example

With a core goal of maximizing research efficiency, process mapping has been a fundamental component of PCRC operations since its inception and has informed performance metric definition and monitoring. We continually examine short and long term goals and progress within the framework of our process maps. Initially, we used the figures generated by Dilts et al (e.g. Figure 1)⁵ and timelines established by cancer cooperative groups to delineate process map components and inform our performance metrics. To do so, we used large sheets of paper and sticky notes to depict algorithms, sequential processes, people involved, and anticipated timelines. We supplemented this information with our prior experience in several large palliative care clinical trials^7-9 and data about times required to accomplish steps in the first PCRC trial.^1,7-10 Using the process map mural as a guide, we maintain a carefully documented timeline of milestones reached, and regularly update our processes and expectations based upon this information. Any instance where a PCRC procedure takes longer to achieve than the benchmarks outlined for the cancer cooperative groups³ or our own internal benchmarks warrants a stakeholder meeting to determine whether or not steps can be streamlined, synchronized, or restructured to increase efficiency. This meeting involves all PCRC team members involved in the process, with ample time to candidly report and critique what is and is not working. The group then brainstorms possible solutions, identifies tasks to improve the process, and delegates responsibility of implementing the changes. Given the complexities and interdependencies involved, it is important for all relevant team members contribute to updating the process as a team.

For example, the creation, review, and finalization of CRFs for our first PCRC study took 4 months longer than projected. CRF development was a single step on the process map, but on closer examination we realized that there were many contributing substeps, each of which carried its own risk for inefficiency. For example, since the protocol had been written and submitted with a grant application prior to the creation of the CRFs, actual operationalization of the study protocol required iterative review cycles regarding the instruments to be used and the wording and order of administration of questions. We did not have an existing set of common data elements from which to pull, so each data element (spanning the spectrum from basic demographic variables to selection of patient reported outcome measures) had to be negotiated and fully defined. In addition, the PCRC did not initially have a standard agreed-upon CRF format. The inefficiencies in CRF development directly impacted study conduct by delaying regulatory submissions and site start up.

In response, the PCRC Statistics and Data Management Coordinating Center thoughtfully reviewed the process and outlined the contributing steps. It was clear that the delays began at the protocol development step and propagated. We held a face-to-face meeting of PCRC operational stakeholders to derive a proposed solution. In collaboration with the Scientific Review Committee we developed a step-by-step procedure of how a protocol is to be internally reviewed by the PCRC. This included having one person — the protocol specialist — responsible for monitoring progress. We outlined who is to review the protocol, expected timeline for reviews, coordination with review committees to ensure concurrent approval of the protocol and CRFs, and a clear understanding of when something was considered “final.” The stakeholder meeting was critical in order to navigate the delicate balance between adding important steps that will improve efficiency and avoiding unnecessary process hurdles and redundant feedback loops. We developed a mapping of PCRC common data elements and are creating a library of common CRFs from current and prior trials in order to reduce development time, increase efficiency, and enhance our ability to conduct cross-trial comparisons. We added all of the new steps to our process map, assisting in the establishment of benchmarks and monitoring progress. In all such decisions we remain mindful of maintaining processes that improve efficiencies and eliminate unnecessary steps.

The goal in developing a research process map is to delineate required steps, eradicate unnecessary steps, rectify identified bottlenecks, and inform iterative updates. Research efficiency is dependent upon streamlined processes that are monitored for compliance, coupled with a commitment to modifying procedures that continuously enhance and improve efficiency. The use of operating procedures and performance metrics provides a structure for achieving our efficiency goals based on the steps delineated in the process map.

Operating Procedures

Operating Procedures are the “go to” documents for researchers at all levels of the organization, including investigators, clinical research coordinators, project managers, data analysts, data managers, monitors, fiscal managers, and those with regulatory roles. The nature and complexity of the research typically dictates the number and types of operating procedures that are necessary. Given the number and diversity of study sites and staff involved in a cooperative research group, the delineation of clear, measurable, and defined tasks that can be implemented consistently and uniformly is paramount in the conduct of rigorous and high quality research. In addition to creating the operating procedures, a research group must identify standard steps for writing, reviewing, and approving the operating procedures (i.e. a “procedure for procedure development”). Specificity of language and the inclusion of relevant examples are keys to effective operating procedures.

Operating procedures, which should be directly tied to the cooperative group process map and metrics, may be divided into two types: Team Operating Procedures (TOPs) and Standard Operating Procedures (SOPs). TOPs, also known as “working instructions” tend to be trial-specific, detailed, and nuanced based on the research protocol. Standard Operating Procedures address more global principles that apply across study sites, trials, and research organizations. SOPs typically define the mission of the organization, outline roles and responsibilities, and delineate the goals of the research. SOPs also establish specific overarching expectations which, when violated, could threaten research validity or participant safety. TOPs are the step-by-step guidelines for completing any study-related task. The most useful TOPs are written with the end user(s) in mind, and provide relevant definitions, examples, roles, exceptions, and references. There are often steps and sub-steps involved in a task. It is important to think through and document these steps so that there is as little room for interpretation as possible.TOPs also become invaluable when training new study personnel.

Practically speaking, SOPs and TOPs can be delineated by whether the procedure defines a process that must happen (i.e., a SOP) versus a guideline of what should happen (i.e., a TOP). Violating a SOP may directly risk study participant safety or threaten the overall validity of the study. SOP violation often is approached with the same level of concern as a protocol violation. SOP violations require clear justification and documentation and can be cause for external audit by a regulatory body or study sponsor. Conversely, TOPs serve as operating guidelines. As such, TOP violations are generally at the Principal Investigator's (or designee's) discretion. While justification and documentation is preferred, a violation of a TOP will not necessarily trigger an external audit. However, multiple violations of TOPs may threaten data quality; close study monitoring is critical to identify and rectify TOP violations.

Operating procedures in action - Palliative care research example

The PCRC started off with a clean slate, in that it had no established operating procedures. The prospect of developing a full palate of SOPs and TOPs was daunting. We thus first, looked to the established cancer cooperative groups and other palliative care research groups for templates and off-the-shelf operating procedures that could be easily adapted. Next, we developed a list of TOPs still needed, prioritized them based on the needs of the first trial, and assigned authors and deadlines. Participant safety and clear guidance to research coordinators interacting with participants were the first priorities. This ensured our overarching goal of collecting high quality data to allow for robust and credible findings. The production of SOPs and TOPs were timed to be in line with various steps in opening and conducting the trial as outlined on our process map.

Many of the PCRC sites have research coordinators who are new to palliative care research, to interventional studies that require participant consent, or both. We thus developed a TOP instructing research coordinators on how to approach potential research participants. The TOP provides guidance on language that allows people with advanced life-limiting illness and their families to feel comfortable when discussing the role of research and whether to participate. Given the myriad conditions and transitions people with advanced illnesses face, the TOP provides key phrases and words to use such as “people like you with advanced illness” as opposed to “people who are dying.” It also addresses how to approach and interact with loved ones involved in the person's care who may fear the idea of research or feel marginalized by the research process. Equipping the research coordinators with “hip pocket words and phrases” like “study” instead of “trial,” “participant” instead of “subject,” and “the family and patient are always in control” adds a level of clarity to the operating procedure that greatly assisted study research coordinators in understanding how to engage potential participants.

Other examples of trial-specific TOPs include “Roles and responsibilities of research team members,” “How to assess performance status using the Australia-modified Karnofsky Performance Status Scale,” and “How to complete case report forms (CRFs).” These are working instructions that are tailored to the specific trial. Each has a clearly delineated audience. For example, the “How to complete case report forms (CRFs)” TOP is designed for the research coordinator. This TOP is written in clear, non-medical language that includes tips and tricks for collecting the CRF information, explains exceptions, and identifies a clear set of instructions should there be a question of interpretation. It takes into account nuances of the study population (e.g., cognitively impaired, children, those with advanced illness) and offers guidance to the research coordinator to ensure that the CRFs are completed uniformly within and across research sites.

While TOPs that provide practical guidance in order to conduct a specific step in a specific trial are critical, we also need to ensure that we have a full portfolio of PCRC SOPs that define expectations for conducting critical cooperative group functions, especially those that span all trials. These SOPs are intended to ensure research integrity and participant safety. Topics include “How to train a new research team member” and “How to obtain informed consent.” Other research concepts that lend themselves to SOPs are issues related to monitoring (e.g., Data Safety Monitoring Board, monitoring compliance with the protocol and operating procedures), human subjects research training, and the storage and protection of data. SOPs do not change frequently regardless of the topic, complexity, or the size of the study. Further, as violations of these rules leave people or the research at risk, a standardized approach is necessary.

Some PCRC processes, such as reporting of adverse and serious adverse events, warrant both a TOP and a SOP. Within the context of the first PCRC study, the adverse event TOP has specific step-by-step instructions for the research coordinator: “If you discover an adverse event, you must complete CRF Form J and submit it to the PCRC data manager in Colorado and your local Institutional Review Board (IRB) within 3 days.” Because adverse event reporting is a regulated task, there also is a need for clear and consistent guidance on the reporting requirements of the local IRB (a.k.a., human research/ethics committee), other regulatory bodies, and funders (a.k.a., sponsors). These are outlined in the corresponding SOP.

TOPs and SOPs must be reviewed and updated at pre-specified intervals and changes to the protocol or regulatory expectations requiring modification must be well documented. For example, in the PCRC's first study there was a substantial protocol amendment that required modifications in the relevant TOP. In the first PCRC study, cognitive impairment was initially an exclusion criterion, but feedback from study sites indicated this was a significant barrier to enrollment. In response, we modified the protocol and consent forms mid-study to include this population. Subsequently, we modified the TOP on obtaining informed consent to include situations in which the patient is cognitively impaired thereby requiring the consent of a legally authorized representative. People with advanced illnesses often experience cognitive impairment (dementia, delirium, or altered consciousness), creating a unique challenge to palliative care researchers trying to include this population in the research. Given the importance of addressing cognitive impairment in the palliative care setting, this TOP will be further refined and generalized into an SOP that will serve as a PCRC-wide approach to obtaining informed consent in this circumstance.

Performance Metrics

Performance metrics, or key performance indicators (KPIs), are necessary in order to evaluate research efficiency. Definition of performance metrics at both the individual study level as well as the cooperative group (or, research group or network) level should be identified early. Performance metrics must closely tie to the process map and operating procedures, must have defined benchmarks, and must be monitored regularly to enhance research efficiency.

Metrics should account for three types of benchmarks: structure, process, and outcome (Table 1), and be defined in terms of both the cooperative group function and conduct of a particular trial within the cooperative group. In the case of individual studies, the metrics for a particular study should be defined as a part of the protocol. Metrics can be represented in terms of proportional conformance to a defined measure (e.g. 65% of studies complete accrual by the pre-specified study end date) or as the numerical result of a particular measure (e.g. time from protocol concept approval to finalized protocol ready for IRB submission is 92 days). Most importantly, the variables informing the measure must be clearly specified so that the characteristics of the measure are easy to operationalize and consistently applied and interpreted. For example, a measure written as “proportion of studies completed on time” allows for broad interpretation of the concept of “complete” and “on time” which may or may not be in line with the intention of the metric.

TABLE 1. Example performance metrics and benchmarks developed for the Palliative Care Research Cooperative Group (PCRC).

Structure – to be assessed for each site and whole PCRC Ability of sites to use a centralized IRB for phase III and non-randomized trials Goal: 90% since centralized IRB functions improve cooperative group efficiency but we don't want to restrict ourselves to only sites that make this allowance Number of sites with experience in palliative care randomized controlled trials Goal: 80%, since we have an objective to always be training some palliative care research naïve sites Availability of a clinical research coordinator at the site experienced in palliative care research Goal: 80%, since we have an objective to always be training some palliative care research naïve sites Process Assessment of the performance of the whole PCRC Time from approval of study concept by the Scientific Review Committee to final version of the study protocol Goal: 12 weeks Time from identification of study-required TOP or SOP to finalization and distribution to study sites Goal: 6 weeks Time from identification of identification of the need to develop a CRF for a protocol to availability for use in the trial Goal: 8 weeks Time from final protocol to first IRB submission Goal: 2 weeks Proportion of site personnel with documentation of annual study-specific training Goal: 100% (training is a SOP imperative) Assessment of the performance of a specific PCRC study (note that summarizing these metrics contributes to assessment of overall cooperative group efficiency) Time from IRB approval to first patient first visit at any site Goal: 4 weeks Time from IRB approval at a site to site training Goal: 2 weeks Time for an invoice to be paid when submitted from a site Goal: 2 weeks Time from data lock to analysis of main results available for manuscript development Goal: 8 weeks Time from availability of main results analysis to final manuscript ready for publication Goal: 8 weeks Assessment of the performance of a specific PCRC site (note that summarizing these metrics contributes to assessment of overall cooperative group and/or study efficiency) Time from PCRC approval of a protocol for IRB submission at a site to site IRB approval Goal: median 12 weeks Time from IRB approval to first patient first visit at the site Goal: 4 weeks Outcomes Assessment of the performance of the whole PCRC Proportion of main study results published within 9 months of study completion Goal: 75% Number of research naïve sites added to PCRC each year Goal: 5 Number of junior investigators supported through the PCRC each year (e.g., funding, access to data for pilot studies) Goal: 5 Proportion of PCRC investigators participating in at least one face-to-face meeting each year Goal: 80% Proportion of annual reports submitted to PCRC sponsor(s) on time Goal: 100% Assessment of the performance of a specific PCRC study (note that summarizing these metrics contributes to assessment of overall cooperative group efficiency) Proportion of accrual target achieved (benchmarked against a running accrual target for each month) Goal: >80% Proportion of CRFs entered into study database within 48 hours of data collection Goal: 95% Impact factor of journal where main results published Goal: >10 Number of papers generated from study data Goal: >5 Assessment of the performance of a specific PCRC site (note that summarizing these metrics contributes to assessment of overall cooperative group and/or study efficiency) Proportion of accrual target achieved (benchmarked against a running accrual target for the site revised each month) Goal: >80% Proportion of CRFs entered into study database within 48 hours of data collection Goal: 95% Proportion of data elements entered generating queries for resolution Goal: <0.05% Proportion of allocated budget distributed to site Goal: >90%

The logical starting point for developing metrics and defining the associated measures is to use the process map and operating procedures. Once measures are identified, it is critical to ensure that the systems are designed to assure that relevant data are readily available and regularly reported. Lack of available data to measure and monitor metrics introduces its own inefficiency, or makes monitoring impossible. There must be an ongoing commitment to measurement and reporting of these metrics, with mechanisms to address deficiencies and refine cooperative group processes and procedures to meet predetermined benchmarks of research efficiency. A standardized metric report should be readily available and regularly distributed to all relevant individuals and especially those in a position to influence this efficiency. The cooperative group should have procedures in place for regularly reviewing these reports and acting on identified opportunities for improvement. Updating reports to real-time dashboards and continuous monitoring tools that place data onto the process map is an evolving concept to optimize research efficiency.

Performance measure in action - Palliative care research example

The PCRC used its evolving process map, TOPs and SOPs to define performance metrics, both for the cooperative group and for our first trial. We brought together investigators and PCRC operations teams to prioritize metrics and decide what we are going to monitor. We recognized in the beginning that it wasn't possible to track and monitor everything well. We elected to prioritize study-related KPIs and some basic cooperative group performance metrics allowing us to refine cooperative group and trial processes and how we track and respond to metrics. We embedded five enrollment-focused KPIs in the study protocol for our first study and sent out recruitment metrics and a graphic summarizing data quality concerns to site investigators monthly (a trial-specific metric). We also monitored time to critical PCRC events such as CRF development, site training and time to first enrolled participant (both cooperative group and study-level metrics). We have expanded our list of PCRC common data elements to accommodate our metrics and are planning a real-time PCRC dashboard to monitor cooperative group and study level efficiency.

In the PCRC, we consider metric development a core task necessary to meet the cooperative group's most fundamental objectives. The process map heralded the importance of site development to expanding PCRC capacity; without site development, studies would not successfully accrue and palliative care research would not be widely generalizable. We are defining site characteristics that will help us describe existing PCRC sites and thoughtfully and strategically add new sites. Characteristics include descriptions of the palliative care populations served (and therefore potential participants in clinical trials) and the types of palliative care and other clinical providers at the site. Investigator's prior research experience, access to research capabilities (e.g. regulatory), palliative care research experience, and methodological interests are also assessed. These elements map directly to the identified characteristics needed to enhance palliative care clinical trials reporting and generalizability of research findings.^11,12 We are envisioning the optimal distribution of sites to continue to build capacity and defining a metric set plus benchmarks that will help us achieve this. We will prospectively monitor these metrics and our portfolio of sites.

Goal Setting and Prioritization

A key, yet underutilized, factor critical to achieve maximum research efficiency is the creation of a measurable goal (or a core set of goals). This is arguably even more important within a complex framework such as that of a cooperative group. Goals in the context of this discussion must be (a) relevant to and (b) embody the success of the program or in this case, the cooperative group. Team involvement in and commitment to setting and achieving these goals is essential. A culture of mutual respect for each member's contributions and skills, an interest in all members being successful, and clear, actionable goals for team members and the enterprise is needed. It is in this sort of environment that invested stakeholders, committed to quality research and ownership of its success are created. Without clear goals, team members may feel directionless, isolated, and possibly undervalued.

To enable successful goal setting and prioritization, one must ensure that a) appropriate time is set aside for the activity; b) it is acceptable to all key members (researchers, staff, trainees/fellows, etc) of the enterprise; and c) it is consistent with the goals of the research enterprise. To maximize buy-in and enhance collaboration, achievement, and efficiencies in research, goals should be developed and delineated into discrete deliverables. To-date, discrete, measurable goals have not been a hallmark of research efficiency initiatives for the PCRC or otherwise. In its next phase, the PCRC plans to address this issue and add it into its framework of success and efficiency.

Monitoring for Success

While defining individual metrics in advance is of prime importance, it is only the first of many steps needed to increase the efficiency of palliative care research. Once metrics are determined, each time point reached during study planning, implementation, analysis, and dissemination needs to be carefully tracked and performance evaluated against the proposed metrics. If the real world time points are consistent with the predicted metrics, it is then reasonable to propose benchmarks to be considered and strict deadlines for each KPI. While some adjustments may need to be made, these benchmarks can likely be adapted and used as a roadmap of time points for similarly structured studies. Having an idea of expected metrics in advance allows palliative care researchers to create a study plan that will improve their ability to reach these benchmarks thus increasing the number of studies that are completed on time and within budget. Regularly scheduled periodic reviews should occur in order to identify and assess any redundant feedback loops and adjust processes as needed.

In addition to regular review, it also is important to remember the goal of monitoring research efficiency. In establishing its core goals in relation to monitoring, the PCRC repeatedly asks its members, “What are we monitoring for?” As a young cooperative group, the PCRC is more intent on promoting quality research than policing for unlikely misuse. At this point in development, the PCRC is focused on a) establishing high quality data and data completeness, b) ensuring there is access to all source documents, c) building each site's research capacity through site trainings and continuing education, and d) fostering and mentoring new investigators. As the PCRC grows in size and confidence with a more diverse research portfolio, the focus of continued monitoring may change, but will always align with each study's extent of risk and the PCRC's overarching commitment to conducting high quality palliative care research as efficiently as possible.

Conclusion

The PCRC has benefitted by learning from the experiences of prior cooperative groups and, as a fledgling cooperative group, has the advantage of purposively considering approaches to maximize research efficiency from its very developmental stages. The PCRC was designed to be nimble. As such, it began with a small membership of committed investigators working in a burgeoning field of research in which there is limited existing evidence. As a new cooperative group, the PCRC is not weighed down by a history of complicated processes or confined due to being part of a complex bureaucratic system. Many key approaches to enhancing research efficiency have been embraced by the PCRC. These include developing a process map early in our development, mapping performance metrics to this process map, goal setting, and continually monitoring and refining our procedures. In addition, the PCRC has actively involved its key stakeholders, investigators and Steering Committee members in these activities, assuring relevant feedback and input.

The PCRC also has been established at a crucial time in the evolution of informatics, the standardization and harmonization of outcomes data, and the sophisticated systems for acquiring, storing, and managing large, multi-site, multi-trial datasets. We are poised to leverage these technological advances and utilize what is available to us (e.g., “off the shelf” electronic data capture systems) in ways that the oncology groups established in the 1950s - 1970s were unable to, rather; they had to build everything from the ground up. These research efficiencies will facilitate achieving the ultimate goal, maximizing return on investment by enhancing care for those with serious illness and their families.

Each of these elements is a core task imperative to good research practice. Although a bit less mature, there should be the same expectations for quality research in the field of palliative care as there are in clinical oncology. Our hope is that the establishment of this expectation up front will help palliative care research to avoid some of the challenges experienced by the cancer cooperatives.

Key Points.

1. The importance of generating meaningful findings and translating these scientific advances to improved patient care creates urgency in the field to address the well-documented system inefficiencies.

2. From its inception, the Palliative Care Research Cooperative Group (PCRC) in the United States was designed to specifically embody the Institute of Medicine's (IOM) vision for future cooperative groups, collaboratives, and networks in terms of running a highly responsive organization that answers cutting edge scientific questions through methodologically-rigorous research studies, the results of which will inform and improve palliative care. Process maps, operating procedures, and performance metrics are the keys to success.

3. Process mapping is key to delineating the steps required to develop a protocol, conduct a study, analyze results, disseminate findings, and manage a cooperative group. It provides the roadmap to improving the performance of a research organization.

4. Given the number and diversity of study sites and staff involved in a cooperative research group, the delineation of clear, measurable, and defined tasks in the operating procedures is essential.

5. Performance metrics must closely tie in to the process map and operating procedures, must have defined benchmarks, and must be monitored regularly to enhance research efficiency.