Strategies for a National Evaluation of the Clinical and Translational Science Awards

A Multidimensional Approach to Evaluation

The Clinical and Translational Science Award (CTSA) Program is the largest initiative funded by the National Center for Research Resources (NCRR) at the National Institutes of Health (NIH). To ensure accountability and a thorough and appropriate evaluation of the effects of an initiative of this scope, the NIH has required that the evaluation be multifaceted, multidimensional, and integrated to an unprecedented degree. Specifically, it has required that the evaluation include local evaluations at each of the 60 award sites, a national evaluation across all 60 sites, and collaboration across sites to identify metrics for each key function and activity undertaken at various levels. Because of the complexity of the award program, the approach to evaluation includes local institutional evaluations, an external program evaluation, and the activities of the Evaluation Key Function Committee, a cross CTSA effort carried out at the CTSA consortium level.

Individual site evaluation

As part of the application for a CTSA, the NIH requires each institution to submit an evaluation plan that includes a logic model delineating its expected activities, outputs, and outcomes. The NIH then requires each institution that received an award to conduct its own internal evaluation of its efforts. Because the evaluations at the local level are internal, they tend to focus on processes and quality improvements.

Even at the local level, the evaluation plan is complex. A CTSA site may have as many as 10 different key functions (essentially equivalent to cores), each with its own activities. In addition to evaluating each of its functions and activities, the site must evaluate its performance and impact as a whole. Because of the multilevel nature of CTSA sites, many explicitly use a systems evaluation approach ¹ , ² that integrates a plan for evaluating CTSA components with a plan for evaluating the CTSA site as a unit.

Each CTSA site engages in various modes of data collection and types of evaluation projects. For example, a site may use social network analysis (SNA) to elucidate relationships among researchers in cross‐cutting disciplines. In some cases, a site may survey its researchers and trainees to assess their level of satisfaction with and success in CTSA‐sponsored activities, or it may survey institution staff, leadership, or other stakeholders to understand issues and challenges regarding the translation of research results into practice. In other cases, a site may use methods such as focus groups, needs assessments, and even town hall meetings to understand issues and challenges.

National evaluation

The NIH has contracted with an outside research and evaluation group to conduct a multifaceted external evaluation of the cross‐CTSA endeavor from a national perspective. This evaluation is summative, rather than formative, and focuses more broadly on changes that the CTSA sites have encouraged and facilitated in the overall clinical and translational research system. For example, the evaluation examines the progress of the CTSA Program in terms of its contribution to strengthening the clinical and translational science workforce via the establishment of CTSA sites, the utilization of CTSA resources by researchers, the implementation of educational and training activities, and the impact of the resources and CTSA activities on institutional and research cultures, on the scientific accomplishments of CTSA researchers, and on translational science as a discipline. Some of the methods used for the national evaluation include surveys of researchers and trainees, bibliometric analyses, and site visits to individual institutions.

Cross‐CTSA objectives and metrics

The individual CTSA sites come together in a networked national consortium that consists of the Executive Committee, the Steering Committee, the Child Health Oversight Committee, 5 Strategic Goal Committees, and 14 Key Function Committees, each of which typically has multiple workgroups with identified projects and activities.

The Evaluation Key Function Committee has formed several workgroups focused on different evaluation issues, approaches, and metrics. For example, the Definitions Workgroup has identified key processes or outcomes that need to be operationally defined for evaluation purposes, and its efforts to understand the concept of translational research have resulted in a publication that introduced a process marker and modeling approach. ³ The workgroup has also been conducting a cross‐site survey of definitions and metrics. The SNA Workgroup has not only examined SNA as a method of evaluation but has also begun conducting a CTSA pilot project evaluation and a community engagement study, both of which use SNA methodology. The Bibliometrics Workgroup has been looking in depth at the methods for bibliometric analysis and has initiated a survey to track all CTSA publications. The Shared Resources Workgroup has conducted annual surveys of the 60 evaluation teams to identify common issues and tools that are emerging.

In addition, members of the Evaluation Key Function Committee have partnered with other CTSA committees on matters pertaining to evaluation. For instance, they worked with the Biostatistics, Epidemiology, and Research Design Key Function Committee to develop standard metrics. ⁴ They helped the Education Key Function Committee identify measures for each factor in the career success model that the committee adopted. ⁵ They worked with the Community Engagement Key Function Committee to develop and publish methods for evaluating community engagement, ⁶ and they helped the Clinical Research Management Key Function Committee develop and conduct several cross‐CTSA studies on the efficiency of institutional review board processes. Moreover, representatives of the Evaluation Key Function Committee have joined with the national evaluation external contractor, the NCRR, and other NIH representatives to form the National Evaluation Liaison Group, whose objective is to encourage coordination and greater efficiency of local and national efforts at CTSA evaluation.

The Future of CTSA Evaluation

The NIH has now reached its target of funding 60 CTSA sites, some of which are in their sixth year of research. Because the CTSA Program is expected to become a major component of the NIH’s proposed National Center for Advancing Translational Science (NCATS), the upcoming evaluations of CTSA sites will be crucial in plotting the future of clinical and translational science research.

The National Evaluation Liaison Group has worked with the American Evaluation Association to draft a white paper that provides an evaluation policy framework for the CTSA sites. In addition, it has worked with the national evaluation external contractor and the NCRR to draft a logic model that creates a shared vision for the CTSA sites during the next 5 years. This model addresses what the group considers to be the most critical research questions related to clinical and translational science and lays the groundwork for an integrated, comprehensive evaluation of efforts to meet CTSA goals. Although the Evaluation Key Function Committee recognizes that the logic model and its associated research questions can continue to be refined, its members have unanimously agreed that the model is consistent with the direction the evaluation should take over the next 5 years.

Conclusion

The complex, multilevel, networked consortium of 60 CTSA sites required an evaluation effort that could assess the varied and multifaceted activities of the individual sites and the consortium as a whole. The evaluation “ecology” that evolved over the first 5 years of the CTSA initiative is well positioned to address the continued requirements of the initiative and to serve as a model for other large‐scale multicenter research initiatives at the NIH and elsewhere.