Abstract
We report the development of the Translational Science Benefits Model (TSBM), a framework designed to support institutional assessment of clinical and translational research outcomes to measure clinical and community health impacts beyond bibliometric measures. The TSBM includes 30 specific and potentially measurable indicators that reflect benefits that accrue from clinical and translational science research such as products, system characteristics, or activities. Development of the TSBM was based on literature review, a modified Delphi method, and in‐house expert panel feedback. Three case studies illustrate the feasibility and face validity of the TSBM for identification of clinical and community health impacts that result from translational science activities. Future plans for the TSBM include further pilot testing and a resource library that will be freely available for evaluators, translational scientists, and academic institutions who wish to implement the TSBM framework in their own evaluation efforts.
Study Highlights
WHAT IS THE CURRENT KNOWLEDGE ON THE TOPIC?
✓ Most current knowledge on the topic of the value of clinical and translational research focuses on productivity measures such as bibliometrics and grant funding.
WHAT QUESTION DID THIS STUDY ADDRESS?
✓ This study addresses the question of how to measure broader impacts of research, including lives saved, improvements to health, and cost savings.
WHAT THIS STUDY ADDS TO OUR KNOWLEDGE
✓ This study presents a framework that can be used to assess or evaluate these processes and outcomes.
HOW THIS MIGHT CHANGE CLINICAL PHARMACOLOGY OR TRANSLATIONAL SCIENCE
✓ By emphasizing the importance of the long‐term impacts of clinical and translational science, we can better understand its value to society and securing funding in an ever‐increasingly competitive budgetary environment.
One reason governments fund research is for the value it provides to society.1 Therefore, it is important to demonstrate what that value is. This can be challenging. Traditional approaches to assess the value of research rely heavily on quantitative measures of scientific productivity, such as grant submissions, grant funding, publications, and citations. This focus on bibliometrics emphasizes the outcomes that are of primary interest to scientists themselves, and overlooks broader benefits to human health and society, such as lives saved, improvements to health, or cost savings. In an era of shrinking research funding and an emphasis to speed translation of research to practice and demonstrate real‐world outcomes that accrue from science, it becomes even more important to develop new approaches for documenting the many ways that research benefits society at large. While this broader impact may be hard to measure, it is critical for understanding the total value our society gains from its investment in the scientific enterprise.2
Following the science into the community is a challenge facing institutions with a Clinical and Translational Science Award (CTSA). CTSAs are funded by the National Center for Advancing Translational Sciences (NCATS). The NCATS goals focus on practices such as training of translational science workforce, increasing the quality and efficiency of translational research, and engaging communities in the translational process.3 The NCATS’ goals are consistent with Woolf's two‐component definition of translational science: i) “the bench‐to‐bedside enterprise of harnessing knowledge from basic sciences to produce new drugs, devices, and treatment options for patients” and; ii) the translation of research into practice; that is, “ensuring that new treatments and research knowledge actually reach the patients or populations for whom they are intended and are implemented correctly.”4 Thus, translational research is fundamentally concerned with how research benefits patients, communities, and populations.
In 2013, the Institute of Medicine (IOM) released an evaluation report on the CTSA program. The report recognized the importance of the CTSA mission for accelerating the translation of basic and clinical science to benefit individual patients and communities. The report also emphasized evaluation and assessment by recommending that NCATS “formalize and standardize the evaluation processes for individual CTSAs and the CTSA Program.”5 Responding to this, NCATS launched the Common Metrics Initiative in December 2015.6 The Initiative aims to develop and implement a set of Common Metrics that will be applied across CTSA Hubs and considered in evaluation of the impact of the CTSA Program as a whole. These metrics cover important measureable objectives focusing on the research process (e.g., median Institutional Review Board (IRB) review duration), career development (e.g., retention of scholars and trainees in clinical and translational research fields, diversity of scholars and trainees), and scientific productivity (e.g., pilot awards that yielded publications and/or subsequent funding).7, 8, 9
These common metrics capture important aspects of the functioning of CTSA programs but are less useful in evaluating longer‐term impacts of translational science. New scientific discoveries from CTSAs are translated into clinical applications, healthcare guidelines, therapeutic techniques, medical devices, community interventions, and other outcomes that benefit societal health. These beneficial outcomes are typically not captured by most scientific impact assessment systems,10 which tend to focus on bibliometric measures of scientific productivity.11 These broader societal and community benefits are difficult to measure for reasons such as the time lag between research discovery and translational applications and difficulty in establishing a direct connection from a specific research output to a subsequent impact. Scientists themselves are often not aware of the impact of their research, lack academic incentives for translation of research to practice, and are not typically trained to communicate their research beyond the scientific community. This type of impact measurement is critical for understanding the total value society gains from its investment in the CTSA awards.
A more systematic method is needed to measure translational scientific endeavors that result in health and social benefits. To address this need, the Institute for Clinical and Translational Sciences (ICTS) Tracking and Evaluation Team (T&E Team) at Washington University in St. Louis developed a new framework for measuring the impact of a large‐scale translational science initiative: the Translational Science Benefits Model (TSBM). The development of the framework was inspired by the desire to look beyond “science begets science” counts of the medical enterprise to identify examples of real‐world clinical and community health impacts that result from translational science activities. The TSBM is informed by recent efforts to develop translational research frameworks and logic models12, 13 and is designed to support institutional assessment of clinical and translational research outcomes. It also can guide future evaluation research designed to identify the most important organizational factors associated with these health and social outcomes.
In this article we i) present the rationale for assessing benefits of translational science beyond scientific productivity; ii) describe the development of the TSBM; and iii) illustrate the application of the TSBM with three case studies.
METHODS
The TSBM is a conceptual framework development study primarily based on a comprehensive literature review, a modified Delphi technique,14 and in‐house expert panel feedback. This project was conducted by the ICTS T&E Team (CTSA Award UL1 TR000448).
We identified models and frameworks for assessing scientific activities, outcomes, and impact through a review of the literature, and compiled a list of domains and indicators that represented stages of clinical and translational science activity. Specific models and frameworks that were reviewed include the Process Marker Model,15 the Translational Research Impact Scale,12 the Research Impact Framework,16 the Payback Model,17 the Snowball Metrics,18 and the Becker Model, previously developed at Washington University.13 The models and frameworks were abstracted and summarized by the authors and members of the ICTS T&E Team.
Indicators of translational science outcomes were then selected from and compared across three sources: the Becker Model for Assessment of Research Impact,13 the Translational Research Impact Scale,12 and the Snowball Metrics.18 These sources were selected because they contained indicators that were specific to clinical and translational science research outcomes, as well as scientific outputs and outcomes. They were combined into a single list of 240 indicators (Supplemental Data). A modified Delphi Method19 was used whereby the ICTS T&E team acted as an expert panel and took several passes through the data to i) remove duplicate items, ii) sort into loose conceptual categories, iii) remove confusing and poorly worded items, and iv) retain items that were the most conceptually clear and most commonly found in the scientific assessment literature. Disagreements among the T&E team were resolved through discussion until consensus was reached.
Given the assessment and evaluation goals of this project, we also developed a logic model for translational science outcomes, based on accepted evaluation best practices.20, 21 This logic model emphasized the translational activities necessary to connect research results and evidence to improved health outcomes.22, 23
The new logic model and translational science benefits indicators were then vetted using an in‐house expert review with the ICTS Director and the Program Director for the ICTS Dissemination and Implementation Research Core. Suggestions were implemented in a revised model, then presented to the members of the ICTS External Advisory Board during an annual meeting. To reflect the focus on clinical and translational benefits, the model was named the Translational Science Benefits Model.
RESULTS
Logic model and benefit domains
The first step in the development of the TSBM was creating a logic model that links scientific activities and outcomes to downstream public health and social benefits (Figure 1). In this model, scientific activities are supported by a variety of financial, infrastructure, human, and knowledge resources. This scientific activity results in a variety of scientific outputs (e.g., results, data, and publications) that in turn are delivered in a variety of translational science settings and products.
Figure 1.
Translational Science Benefits Logic Model.
Although the TSBM is consistent with broader models of scientific activity,16, 24, 25 it was designed to be specifically applicable to clinical and translational research. To that end, two translational phases are highlighted in the model. First, the raw results of scientific activities are translated for a primarily scientific audience. This captures the traditional flow of science into scientific journals, conferences, and other venues where scientific results become visible and primarily influence other scientists.
The second translational phase is the process of dissemination and implementation where scientific findings that are relevant for new treatments, programs, and policies are tailored and disseminated to a broader audience of clinicians, hospital administrators, policymakers, health advocates, and funders.
The primary contribution of the TSBM is to identify and define the areas where clinical and translational science provides health and societal benefits. We suggest that these benefits fall into four categories. i) Clinical and Medical Benefits: Adoption and implementation of new tools and procedures in clinical settings as a result of clinical and translational research. ii) Community and Public Health Benefits: Enhancement of healthcare or community and population well‐being as a result of clinical and translational research. iii) Economic Benefits: Economic, commercial, or financial improvements that result from clinical and translational research. iv) Policy and Legislative Benefits: Involvement with the policy‐making process or formal adoption into organizational or public policies, legislation, or governmental standards based on clinical and translational research. This definition is broad enough to include both “big P” policies (formal laws or rules enacted by governmental bodies) and “small P” policies (organizational guidelines, internal agency decisions).26
More specifically, the TSBM suggests that certain results and discoveries made by clinical and translational scientists get translated and disseminated beyond a purely scientific audience. The fruits of this science may lead to advances and innovations in clinical settings; changes to health systems and health promotion activities that lead to health improvements in communities and populations; development of commercial products and other advances that have positive economic consequence; and finally, implementation of evidence‐based policies and regulations that all serve to reduce disease and disability, and promote public health. Figure 2 lists the four benefits domains with their indicators. Each of the domains includes two or three categories that group the indicators into similar sets. For example, five of the Clinical & Medical Benefits indicators are all types of tools and products.
Figure 2.
Translational Science Benefits Model Domains and Indicators.
The final TSBM includes 30 indicators and their definitions (Appendix). The definitions are relatively straightforward, but the National Library of Medicine Medical Subject Headings (MeSH) was also used to enhance consistency across definitions.27 These indicators reflect the wide variety of benefits that accrue from clinical and translational science research while including specific things or products (e.g., new biomedical technology used in a clinical setting), system characteristics or activities (e.g., accessibility of healthcare services), or effects (e.g., improved life expectancy). The indicators are specific, observable, and are each at least potentially measurable. That is, they are defined in such a way that they can lead to a measurable metric, and can be linked to institutional SMART objectives (i.e., specific, measureable, achievable, relevant, time‐bound).28 For example, if a translational research project leads to development of a new electronic technology for healthcare providers to use in the clinical setting, this benefit would be captured by the Software Technologies indicator within the Clinical and Medical Benefits domain. Conversely, if a research program resulted in updated evidence‐based guidelines by a public health body such as the Centers for Disease Control, that benefit would be captured by the Public Health Practices indicator in the Community and Public Health Benefits domain. In this sense, the TSBM indicators represent new or improved benefits that accrue from clinical and translational science research.
Case studies
To provide supporting information regarding the feasibility and face validity of the TSBM, we conducted a set of case studies demonstrating the application of the TSBM to clinical and translational science research projects that were led by Washington University ICTS members.29 To determine potential projects for case studies, the ICTS leadership identified bench and clinical investigators who had received ICTS core services or funding from 2007–2012. Studies were selected by evaluating ICTS core usage, publication, grant application, and award data while considering anecdotal information such as success stories highlighted in institutional, regional or national media and feedback to ICTS leadership about research findings. We do not expect that any single clinical research study would lead to all the benefits described in the TSBM. However, this type of research often leads to benefits in more than one of the four TSBM domains (e.g., a clinical study producing a new diagnostic tool could very well lead to clinical benefits as well as economic benefits). To that end, the case study development emphasized clinical research that could have benefits in more than one domain.
Ten research studies were identified as possibilities to use for preliminary testing of the new model. Of those, three studies whose findings resulted in potential scientific and translational benefits were ultimately selected for case study development. These three studies were selected because i) they represented at least one basic/bench science study and one clinical research study, ii) they represented translational research in different stages, iii) they illustrated a variety of indicators from the four TSBM domains, and iv) their investigators were willing to participate in the development of the TSBM. These case studies represent preliminary testing results for illustrative purposes and identifying challenges for the future development of an assessment system based on the TSBM.
CASE STUDY #1: SPECIFIC NON‐INVASIVE DIAGNOSIS OF KIDNEY CANCER
Study description
The Specific Non‐Invasive Diagnosis of Kidney Cancer study explores urine biomarkers for detecting kidney cancer. Jeremiah J. Morrissey, PhD, received ICTS pilot funding in 2009 and 2011 to study urine biomarkers. The goals of this study were to establish the ability of markers in urine and blood to diagnose kidney cancer, differentiate kidney cancers from other cancers of the urinary tract or common noncancerous kidney diseases, and monitor for recurrence and the effectiveness of chemotherapy in patients with metastatic disease.
Dr. Morrissey reported that the findings from the study validated the clinical utility of the urine biomarkers for screening of renal cell carcinoma.45, 46 His team is currently investigating several noninvasive screening test applications for detecting kidney cancer at early, more treatable stages before patients have symptoms. The study is noteworthy as there is currently no method for early detection of kidney cancer, nor are there methods of surveillance of recurrence or testing of response to chemotherapy. These challenges are important to overcome because kidney cancer is difficult to treat unless caught early and is known to be resistant to chemotherapy and radiation.
TSBM indicators
Early benefits of the Kidney Cancer study have been observed in two of the four TSBM domains over the past few years (Table 1). A patent for methods to detect and diagnose renal cancer using biomarkers has been granted and is available for licensing.30 Studies for the development of screening assays for early detection of kidney cancer are in process. Development of promising and affordable applications for early detection is expected to provide several benefits: reduced mortality rate, less invasive treatment, and preservation of renal function. Early detection methods for kidney cancer can be incorporated into everyday medical practice and provide significant benefits to societal health.
Table 1.
Translational Science Benefits Model observed indicators for the Diagnosis of Kidney Cancer study
| Benefit | TSBM Domain | TSBM Indicator |
|---|---|---|
| Early screening test for kidney cancer studies are in process; | Clinical and Medical Benefits | Diagnostic Procedures |
| Patent application enhances the chance of potential commercialization of the test. | Economic Benefits | Patents |
CASE STUDY #2: DIAGNOSTIC PROCEDURES FOR CREUTZFELDT–JAKOB DISEASE (CJD)
Study description
The Diagnostic Procedures for Creutzfeldt–Jakob Disease (CJD) study explores diffusion tensor imaging (DTI) as an early biomarker for diagnostic purposes. CJD is a rapidly progressive neurodegenerative disease (RPD) with diagnosis often made at autopsy. Beau M. Ances, MD, PhD, MSc, was awarded ICTS pilot funding in 2009 for research to study CJD and DTI. The goal of this study was to identify early changes in the brain structure due to CJD that may allow for early intervention. The research led to hosting of the Evaluation of CJD & other Rapidly Progressive Dementias conference in St. Louis, a CJD Foundation Family Workshop in 2012, and several journal articles.31, 32
The CJD research was important because it resulted in new preliminary diagnostic procedures and clinical practices and increased the community awareness of a rare and devastating disease. The results have led to procedures for earlier and less invasive diagnosis of CJD, benefiting both patients and healthcare workers.
TSBM indicators
Benefits of the CJD study represent all four TSBM domains (Table 2). Under the Clinical and Medical Benefits domain, the CJD study has led to a new Barnes‐Jewish Hospital policy for treatment of CJD patients and resulted in preliminary identification of clinical and diagnostic tests such as MRI and lumbar puncture to distinguish CJD from other RPDs. For the Community and Public Health Benefits domain, the CJD study saves patients from the need for brain biopsy, reduces risks to patients and healthcare workers, and received recognition for the CJD Foundation as a source of knowledge and assistance for patients and families, which has led to increased referral and volume of CJD patients to Washington University. For the Economic Benefits domain, fewer tests are being ordered for patients and the length of hospital stay has decreased. For the Policy and Legislative Benefits domain, results of the CJD study have been communicated to State Health Boards with revisions in policies leading to an increase in reporting of CJD cases in Texas. These outcomes reflect tangible real‐world health impacts.
Table 2.
Translational Science Benefits Model observed indicators for the Creutzfeldt‐Jakob Disease study
| Benefit | TSBM Domain | TSBM Indicator |
|---|---|---|
| Preliminary identification of clinical and diagnostic tests to distinguish CJD from other RPDs; | Clinical and Medical Benefits | Diagnostic Procedures |
| Saves patients from need for brain biopsy; reduces risks to patients and healthcare workers; | Community and Public Health Benefits | Healthcare Delivery |
| Recognition from the CJD Foundation as source of knowledge and assistance for patients and families; increased referral of CJD patients to Washington University; | Community and Public Health Benefits | Healthcare Accessibility |
| Fewer tests ordered; decreased length of hospital stay; | Economic Benefits | Cost Effectiveness |
| Change in Barnes‐Jewish Hospital policy regarding treatment of CJD patients; | Policy and Legislative Benefits | Policies |
| Change in Texas State policy for diagnosis of CJD. | Policy and Legislative Benefits | Policies |
CASE STUDY #3: THE CONTRACEPTIVE CHOICE PROJECT
Study description
The Contraceptive CHOICE Project is a large clinical trial research study with the goal of increasing uptake of long‐acting reversible contraception and decreasing unintended pregnancy in the St. Louis area. Jeffrey Peipert, MD, PhD, a Principal Investigator for the Contraceptive Choice Project, utilized support from the ICTS from 2007 to 2012 for KL2 scholars, TL1 trainees, and consultations with ICTS related to this project. The project involved nearly 10,000 women from the area along with community partners and private providers.33 Providing no‐cost contraception to teens in the CHOICE Project dramatically reduced the teen pregnancy and abortion rate for the St. Louis area. Of the 1,404 teens in the project, 72% chose a Long‐Acting Reversible Contraceptive (LARC) method. The teen pregnancy rate was 34.0 per 1,000 teens compared to the national average of 158.5 per 1,000 teens. Additionally, the abortion rate for teens in the CHOICE project was 9.7 per 1,000 teens compared to the national average of 41.5 per 1,000 teens.34 Reduction of teen pregnancy rates is among the six identified CDC Winnable Battles for 2010–2015.35
The CHOICE Project is important from a clinical and translational aspect, as it represents a fundamental shift in the use of contraceptive methods and garnered significant attention from the community and media. The project demonstrated that removing barriers to highly effective contraceptive methods such as IUDs and implants reduces unintended pregnancies and the need for abortions.
TSBM indicators
Benefits of the CHOICE Project represent three of the four TSBM domains (Table 3). As a Clinical and Medical Benefit, CHOICE was cited in two professional pediatric clinical guidelines: the American Academy of Pediatrics Policy Statement, Contraception for Adolescents, 2014, and the American College of Obstetricians and Gynecologists Practice Bulletin, Long‐Acting Reversible Contraception: Implants and Intrauterine Devices, 2013.36, 37
Table 3.
Translational Science Benefits Model observed indicators for the Contraceptive CHOICE Project
| Benefit | TSBM Domain | TSBM Indicator |
|---|---|---|
| CHOICE Project cited in two evidence‐based guidelines; | Clinical and Medical Benefits | Guidelines |
| New women's community health center (C3) established, based on the Contraceptive Choice Project model of care; | Community and Public Health Benefits | Community Health Services |
| Project members have provided training to ACOG, CDC, Association of Reproductive Health Professionals, CHOICE‐Australia; | Community and Public Health Benefits | Healthcare Delivery |
| Local teen pregnancy rates and abortion rates were significantly below national average; | Community and Public Health Benefits | Life Expectancy and Quality of Life |
| Cited in U.S. Supreme Court Hobby Lobby case. | Policy and Legislative Benefits | Expert Testimony |
Community and public health benefits of the CHOICE Project are seen in three different ways: a new women's health community center (C3) based on the Contraceptive Choice Project model of care has been established,38 project investigators currently provide clinical training for members of the community, and the recommendations from the study are currently being tested in Australia (The Australian Contraceptive Choice Project, ACCoRd).39 The clinical trial itself dramatically reduced unintended pregnancy and abortion rates for the entire St. Louis region. Finally, the CHOICE project has also shown potential policy impacts since it was cited in an amicus brief submitted to the U.S. Supreme Court in 2014 by the Guttmacher Institute.40 The CHOICE project was cited as evidence of how effective contraception reduces the need for abortion, potentially preventing more than half of abortions performed annually.
THE CASE STUDIES AND THE STAGES OF TRANSLATIONAL RESEARCH
It is important to clarify that the case studies explored here were in different phases and represent different stages of translational research. Each of these points affects the analysis and classification of translational science benefits. Figure 3 places the case studies on a continuum of translational research.41 Cases 1 (Kidney) and 2 (CJD) concerned identification of problems and development of interventions (T0‐T1). Case 2 also developed guidelines for early detection of CJD and led to policy and process changes through implementation of the new guidelines (T2‐T3). Finally, Case 3 (CHOICE) began at a later translational research stage by testing an evidence‐based intervention (T3) and evaluating its effectiveness in the real world (T4).
Figure 3.
Case Studies and the Stages of Translational Research.
DISCUSSION
In this article we presented a new framework for identifying and describing the many benefits that can derive from translational and clinical science. The TSBM is based on past work from the field of scientific evaluation, but focuses on translational science and is designed to support feasible evaluation metrics and processes. As the three case studies suggest, successful clinical research that is primarily designed to answer important biological, medical, or healthcare delivery questions can have long‐term effects (Table 4). In just a few years after initial funding and support from CTSA resources, these studies resulted in clinical, community, economic, and policy benefits that analysis of publication, grant application, or survey data would not have revealed. Furthermore, placing these studies in the larger framework of the stages of translational research suggests that as a research program progresses through these stages, benefits will likely increase, and benefits from research at later stages may be more readily achievable or apparent than those at earlier ones, i.e., the two studies addressing T2‐T4 research garnered more tangible benefits.
Table 4.
Summary of Translational Science Benefits Model indicators for three case studies
| Health & Societal Benefits | Case Study #1 Kidney | Case Study #2 CJD | Case Study #3 CHOICE |
|---|---|---|---|
| Clinical and Medical Benefits | |||
| Tools and Products | |||
| Procedures and Guidelines | ✓ | ✓ | ✓ |
| Community and Public Health Benefits | |||
| Health Activities and Products | ✓ | ||
| Healthcare Characteristics | ✓ | ✓ | |
| Health Promotion | ✓ | ||
| Economic Benefits | |||
| Financial Savings and Benefits | ✓ | ||
| Commercial Products | ✓ | ||
| Policy and Legislative Benefits | |||
| Advisory Activities | ✓ | ||
| Policies and Legislation | ✓ |
Measuring the benefit of research is an emerging methodology and requires innovative tools to supplement peer‐review and bibliometrics.42 Measuring benefits that fall outside of scientific productivity metrics have their own set of challenges. A distinct community devoted to evaluation of research benefits with its own series of conferences or journals does not currently exist.43 Although some benefits such as formal clinical guidelines or patents have existing administrative methods for tracking, many others do not (e.g., expert testimony). Finally, even with new health and societal benefit metrics available, we are still confronted with the basic evaluation challenges of how to demonstrate direct associations between research funding, scientific outcomes, and subsequent downstream social and health benefits.44 Implementing formal evaluation systems based on the TSBM requires new partnerships between clinical scientists, academic administration, funders, and many relevant stakeholders (e.g., practitioners, policy makers).
The TSBM complements other efforts to study the processes and outcomes of research funding. Universities and research organizations are striving to translate benefits beyond academia to demonstrate impactful research outcomes and contributions towards the mission of the university. For example, the American Association of Medical Colleges (AAMC), with the support of RAND Europe, launched a research evaluation initiative to help medical schools introduce more comprehensive approaches to evaluating the value of research and provide a fuller picture in which research benefits institutions, patients, and communities.47 The three areas of focus are clinical outcomes improvement research, health equity research, and basic research.
Funding agencies are also focusing on metrics that transcend traditional measures of reporting on productivity (e.g., citations, journal impact factor score) towards outcomes such as clinical applications, influence on public policy, or community engagement endeavors. One recent effort is the Science Impact Framework developed by the Centers for Disease Control and Prevention (CDC), employing indicators to measure impact towards health outcomes.48 Agencies such as the National Institute of Environmental Health Sciences (NIEHS) have implemented strong evaluation programs that emphasize reporting of qualitative‐based outcomes and produced a manual designed to demonstrate achievements in environmental public health.49
FUTURE PLANS
The long‐term goal for the TSBM project is to develop the model as an evaluation system for clinical and translational research to supplement assessment systems that use traditional bibliometric methods. The activities reported here represent the first part of this longer‐term project. Two more steps are planned that will build off the developed TSBM logic model and indicators. First, a TSBM resource library is being developed that will provide guidance on how to use each indicator. This is being developed by ICTS and the Becker Medical Library at Washington University, and will be freely available for clinical and translational scientific institutions (CTSA Hubs and others). Second, pilot‐testing of specific metrics that can be used to measure each of the TSBM indicators will be conducted. This will provide important guidance for evaluators, translational scientists, and academic institutions who wish to implement the TSBM in their own evaluation efforts.
Conflict of Interest
The authors declared no conflicts of interest.
Supporting information
Supporting Information
Supporting Information
Acknowledgments
The authors thank Drs. Ances, Morrissey, and Peipert for their assistance with the TSBM project and providing the data for the case studies. The authors also thank Laura Brossart for assistance with model layout and appearance.
The research reported in this publication was supported by the Washington University Institute of Clinical and Translational Sciences grant UL1 TR000448 from the National Center for Advancing Translational Sciences (NCATS) of the National Institutes of Health (NIH). The content is solely the responsibility of the authors and does not necessarily represent the official view of the NIH.
Author Contributions
D.A.L., C.C.S., A.M.S., B.J.C., T.B.C., J.L.A., C.E.B., and B.A.E. wrote the article. D.A.L., C.C.S., A.M.S., B.J.C., T.B.C., J.L.A., and B.A.E. designed the research. D.A.L., C.C.S., and A.M.S. performed the research. C.C.S. and A.M.S. analyzed the data.
References
- 1. Smith, S. , Ward, V. & House, A. ‘Impact’ in the proposals for the UK's Research Excellence Framework: Shifting the boundaries of academic autonomy. Res. Policy 40, 1369–1379 (2011). [Google Scholar]
- 2. Donovan, C. State of the art in assessing research impact: introduction to a special issue. Res. Eval. 20, 175–179 (2011). [Google Scholar]
- 3. National Institutes of Health . National Center for Advancing Translational Sciences (NCATS). [cited 2015. Jul 28]. Available from: https://ncats.nih.gov/ctsa/about [Google Scholar]
- 4. Woolf, S.H. The meaning of translational research and why it matters. JAMA 299, 211–213 (2008). [DOI] [PubMed] [Google Scholar]
- 5. Institute of Medicine . The CTSA Program at NIH: Opportunities for advancing clinical and translational research. (National Academies Press, Washington, DC, 2013). [PubMed] [Google Scholar]
- 6. CTSA Clinical and Translational Science Awards . Common Metrics (Draft). Presented at: CTSA Program Annual PI Meeting 2015 Dec 16–17; Washington DC. [cited 2016 Oct 5]. Available from: https://ctsacentral.org/wp-content/uploads/CTSA-Common-Metrics-Booklet-12-16-2015.pdf
- 7. Rubio, D.M. Common metrics to assess the efficiency of clinical research. Eval. Health Prof. 36, 432–446 (2013). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 8. Rubio, D.M. et al Developing common metrics for the Clinical and Translational Science Awards (CTSAs): lessons learned. Clin. Transl. Sci. 8, 451–459 (2015). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 9. Tufts Clinical and Translational Science Institute . Common Metrics Initiative [Internet]. 2016. [cited 2016 Oct 5]. Available from: http://www.tuftsctsi.org/research-services/research-process-improvement/common-metrics-initiative/
- 10. Searles, A. et al An approach to measuring and encouraging research translation and research impact. Health Res. Policy Syst. 14, 1–13 (2016). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 11. Louis, C.F. & Reed, G. The current state and recommendations for meaningful academic research metrics among American research universities Research Metrics Working Group, US Research Universities Futures Consortium (2013). [Google Scholar]
- 12. Dembe, A.E. , Lynch, M.S. , Gugiu, P.C. & Jackson, R.D. The translational research impact scale: development, construct validity, and reliability testing. Eval. Health Prof. 37, 50–70 (2014). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 13. Sarli, C.C. , Dubinsky, E.K. & Holmes, K.L. Beyond citation analysis: a model for assessment of research impact. J. Med. Libr. Assoc. 98, 17–23 (2010). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 14. Hsu, C.C. & Sandford, B.A. The Delphi Technique: Making sense of consensus. Pract. Assess. Res. & Eval. 12, 1–8 (2007). [Google Scholar]
- 15. Trochim, W. , Kane, C. , Graham, M.J. & Pincus, H.A. Evaluating translational research: a process marker model. Clin. Transl. Sci. 4, 153–162 (2011). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 16. Kuruvilla, S. , Mays, N. , Pleasant, A. & Walt, G. Describing the impact of health research: a research impact framework. BMC Health Serv. Res. 6, 134 (2006). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 17. Buxton, M. & Hanney, S. How can payback from health services research be assessed? J. Health Serv. Res. Policy 1, 35–43 (1996). [PubMed] [Google Scholar]
- 18. Colledge, L. Snowball Metrics Recipe Book. 2nd ed (2014). Available from: https://www.snowballmetrics.com/wp-content/uploads/snowball-recipe-book_HR.pdf [Google Scholar]
- 19. Dalkey, N. The Delphi method: an experimental study of group opinion. Futures 1, 408–426 (1969). [Google Scholar]
- 20. Cooksy, L.J. , Gill, P. & Kelly, P.A. The program logic model as an integrative framework for a multimethod evaluation. Eval. Program Plann. 24, 119–128 (2001). [Google Scholar]
- 21. W.K. Kellogg Foundation . Using Logic Models to Bring Together Planning, Evaluation, and Action: Logic Model Development Guide. 2004. [cited 2016 Oct 5]. Available from: http://www.wkkf.org/resource-directory/resource/2006/02/wk-kellogg-foundation-logic-model-development-guide
- 22. Glasziou, P. & Haynes, B. The paths from research to improved health outcomes. Evid. Based Nurs. 8, 36–38 (2005). [DOI] [PubMed] [Google Scholar]
- 23. Brownson, R.C. , Colditz, G.A. & Proctor, E.K. Dissemination and Implementation Research in Health: Translating Science to Practice. (Oxford University Press, New York, 2012). [Google Scholar]
- 24. Hanney, S.R. , Grant, J. , Wooding, S. & Buxton, M.J. Proposed methods for reviewing the outcomes of health research: the impact of funding by the UK's “Arthritis Research Campaign.” Health Res. Policy Syst. 2, (2004). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 25. Weiss, A.P. Measuring the impact of medical research: moving from outputs to outcomes. Am. J. Psychiatry 164, 206–214 (2007). [DOI] [PubMed] [Google Scholar]
- 26. Brownson, R. C. , Seiler, R. & Eyler, A. A. Measuring the impact of public health policy. Prev. Chronic Dis. 7, A77 (2010). [PMC free article] [PubMed] [Google Scholar]
- 27. National Institutes of Health . U.S. National Library of Medicine Medical Subject Headings. 1999. [updated 2016 Nov 16; cited 2016 Oct 5]. Available from: https://www.nlm.nih.gov/mesh/
- 28. Doran, G.T. There's a S.M.A.R.T. way to write management's goals and objectives. Manage. Rev. 70, 35–36 (1981). [Google Scholar]
- 29. Guthrie, S. , Wamae, W. , Diepeveen, S. , Wooding, S. & Grant, J. Measuring Research: A Guide to Research Evaluation Frameworks and Tools. (RAND Corporation, Santa Monica, CA, 2013). Available from: http://www.rand.org/pubs/monographs/MG1217.html [Google Scholar]
- 30. Kharasch, E. D. & Morrissey, J. J. , inventors; Washington University, assignee . Methods of renal cancer development. United States patent US 9,091,690. (2015).
- 31. Wang, L. H. et al Role of magnetic resonance imaging, cerebrospinal fluid, and electroencephalogram in diagnosis of sporadic Creutzfeldt‐Jakob disease. J. Neurol. 260, 498–506 (2013). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 32. Kang, P. et al Sleep pathology in Creutzfeldt‐Jakob disease. J. Clin. Sleep Med. 12, 1033–1039 (2016). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 33. McNicholas, C. , Madden, T. , Secura, G. & Peipert, J. F. The contraceptive CHOICE project round up: what we did and what we learned. Clin. Obstet. Gynecol. 57, 635–643 (2014). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 34. Secura, G.M. et al Provision of no‐cost, long‐acting contraception and teenage pregnancy. N Engl J Med. 371, 1316–1323 (2014). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 35. Centers for Disease Control and Prevention . Winnable Battles: Teen Pregnancy. [updated 2014 Nov 3; cited 2016 Oct 5]. Available from: http://www.cdc.gov/winnablebattles/teenpregnancy/index.html
- 36. American Academy of Pediatrics . Contraception for adolescents. Pediatrics 134, e1244–e1256 (2014). [DOI] [PubMed] [Google Scholar]
- 37. American College of Obstetricians and Gynecologists . Practice Bulletin No. 121: Long‐acting reversible contraception: implants and intrauterine devices. Obstet. Gynecol. 118, 184–196 (2011). [DOI] [PubMed] [Google Scholar]
- 38. Washington University in St. Louis . New women's health clinic at Washington University [Internet]. Wellness Connect 2015 Oct. 6. [cited 2016 Nov 1]. Available from: http://wellnessconnection.wustl.edu/new-womens-health-clinic-at-washington-university/
- 39. University of Technology Sydney . Increasing the use of long‐acting reversible contraception: The Australian Contraceptive ChOice pRoject (ACCORd) [Internet]. 2015 [cited 2016 Nov 1]. Available from: http://cfsites1.uts.edu.au/find/projects/detail.cfm?ProjectId = 2015000144
- 40. Brief of the Guttmacher Institute and Professor Sara Rosenbaum as Amici Curiae in support of the Government, Burwell v. Hobby Lobby, 134 S. Ct. 2751, 573 US__189 (2014), (Nos. 13–354 & 13–356).
- 41. Glasgow, R.E. , Vinson, C. , Chambers, D. , Khoury, M.J. , Kaplan, R.M. & Hunter C. National Institutes of Health approaches to dissemination and implementation science: current and future directions. Am. J Public Health 102, 1274–1281 (2012). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 42. Hazelkorn, E. Measuring Value: Societal Benefits of Research [Internet]. Chronicle of Higher Education 2012. Aug 15 [cited 2016 Oct 5]. Available from: http://www.chronicle.com/blogs/worldwise/measuring-value-societal-benefits-of-research/30179 [Google Scholar]
- 43. Bornmann, L. Measuring the societal impact of research. EMBO Rep. 13, 673–676 (2012). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 44. Mayne, J. Addressing attribution through contribution analysis: Using performance measures sensibly. Can. J. Progr. Eval. 16, 1–24 (2001). [Google Scholar]
- 45. Morrissey, J.J. & Kharasch, E.D. The specificity of urinary Aquaporin-1 and Perilipin-2 to screen for renal cell carcinoma. J. Urol. 189, 1913–1920 (2013). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 46. Morrissey, J.J. et al. Evaluation of urine Aquaporin-1 and Perilipin-2 concentrations as biomarkers to screen for renal cell carcinoma: A prospective cohort study. JAMA Oncol. 1, 204–212 (2015). [DOI] [PMC free article] [PubMed] [Google Scholar]
- 47. RAND Europe . AAMC research evaluation initiative: Panel workshop report. 1–5 (2013). Available from: https://www.aamc.org/download/400200/data/crosspanelreport.pdf
- 48. Centers for Disease Control and Prevention . Description of the Science Impact Framework [Internet]. 2014. [updated 2016 Feb 29; cited 2015 Oct 20]. Available from: https://www.cdc.gov/od/science/impact/framework.html
- 49. National Institutes of Environmental Sciences . Partnerships for Environmental Public Health Evaluation Metrics Manual. NIH Publication No. 12–7825. 2012. Available from: http://www.niehs.nih.gov/research/supported/translational/peph/metrics/index.cfm
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.
Supplementary Materials
Supporting Information
Supporting Information