Abstract
Knowledge translation is at the epicenter of 21st century life sciences and integrative biology. Several innovative institutional designs have been formulated to cultivate knowledge translation. One of these organizational innovations has been the Center for Translational Molecular Medicine (CTMM), a multi-million public–private partnership in the Netherlands. The CTMM aims to accelerate molecular diagnostics and imaging technologies to forecast disease susceptibilities in healthy populations and early diagnosis and personalized treatment of patients. This research evaluated CTMM's impact on scientific, translational, clinical, and economic dimensions. A pragmatic, operationally-defined process indicators approach was used. Data were gathered from CTMM administrations, through a CTMM-wide survey (n = 167) and group interviews. We found that the CTMM focused on disease areas with high human, clinical, and economic burden to society (i.e., oncology, cardiovascular, neurologic, infection, and immunity diseases). CTMM displayed a robust scientific impact that rests 15%–80% above international reference values regarding publication volume and impact. Technology translation to the clinic was accelerated, with >50% of projects progressing from pre-clinical development to clinical testing within 5 years. Furthermore, CTMM has generated nearly 1500 Full Time Equivalent (FTE) of translational R&D capacity. Its positive impact on translational, (future) clinical, and economic aspects is recognized across all surveyed stakeholders. As organizational innovation is increasingly considered critical to forge linkages between life sciences discoveries and innovation-in-society, lessons learned from this study may inform other institutions with similar objectives such as the Clinical and Translational Science Awards (CTSA) Program of the National Institutes of Health (NIH) in the United States.
Introduction
Investment in biomedical research has fueled the development of new medical technologies (i.e., drugs, diagnostic methods, drug delivery systems, and medical devices). Herewith, astonishing medical advances have been achieved, with stimulation of high-wage job growth and economic health.
Public investments have been an important enabler of biomedical research and development (R&D). Between 2007 and 2012, the United States contributed the largest share of total global public sector expenditures (circa 52%), followed by Europe (circa 27%) and Asia-Oceania (circa 18%). The private sector has even shouldered a larger proportion of biomedical R&D expenditure, on balance approximately 64% of overall spending (Chakma et al., 2014).
In order for societies to enjoy the expected benefits of these investments fully, biomedical research findings have to be moved from “bench to bedside” and into the community (Rubio et al., 2010). Motivated by the assertion that a time lag of circa 17 years for research evidence to reach clinical practice is unnecessarily long, the concept of translational research has attracted large interest from both public and private parties (Balas and Boren, 2008; Contopoulous-Ioannidis et al.; 2008, Trochim et al.; 2011; Westfall et al., 2007).
Various countries established programs to incentivize translational research. A widely known example is the Clinical and Translational Science Awards (CTSA) Program of the National Institutes of Health (NIH) in the US. In the Netherlands, the Center for Translational Molecular Medicine (CTMM) was established in 2008. It was born out of recognition by academia, industry, and the Dutch government that translating fundamental research into patient benefit and economic activity requires a truly multidisciplinary end-to-end approach that involves all parties in the value chain. The public–private partnership, including universities, academic hospitals, and pharmaceutical and medical technology companies, aims to accelerate molecular diagnostics and imaging technologies to enable determination of predisposition, early diagnosis, and personalized treatment of patients.
Ultimately, CTMM aspires to reduce mortality, morbidity, as well as health care costs significantly, by funding translational research projects in high-burden disease areas. With a total research budget of circa €300 million, including 50% government, 25% university, and 25% commercial funding, CTMM was one of the first and largest public–private biomedical research partnerships internationally. Herewith the need arose to assess its impact. The key findings regarding CTMM's scientific, translational, clinical, and economic impact, as achieved 5 years after becoming operational in 2008, are described in this report.
Methods
Study setting
At the beginning of 2012, CTMM had brought together 124 active project partners, including 27 academic institutes and 97 industrial partners. Figure 1 shows the development of CTMM's partner portfolio over time. Notably, CTMM has attracted a relatively large number of Small and Medium Enterprises (SMEs) from the start, up to 74 SMEs in year 2012. Partners are clustered into project consortia, with circa 9–11 partners per project, and are supplied with the resources and infrastructures needed to achieve results collectively. The number of projects increased from 7 in year 2008 to 22 in year 2012 (Fig. 2). Projects examined in this report have had a timeframe of 4–5 years and were finalized no later than 2015.
FIG. 1.

Development of CTMM Partner Portfolio 2008–2012.
FIG. 2.
Number of CTMM projects by year and disease area.
CTMM projects focus on four disease areas that are associated with high human burden and high societal (oncology, cardiovascular, neurodegenerative, infection, and immunity-related diseases). In addition, an overarching Information Technology (IT) project was established in October 2011, to build a long-lasting IT infrastructure for the Netherlands to facilitate the collection, storage, analysis, archiving, and securing of the vast amounts of data generated in CTMM's scientific program.
Study design
The research impact analysis of CTMM was commissioned in 2012, nearly 4 years after its start in 2008. As such, the general approach and design of the research impact analysis were necessarily ad hoc, guided by administrative data and output metrics that had been collected thus far by CTMM and were complemented by a cross-sectional survey among the CTMM participants.
While it is widely accepted that the endpoint of translational research lies ultimately in improved health outcomes and other societal impacts, the planned duration of CTMM as a public–private partnership, as well as the feasible time-horizon of the research impact analysis, were too short to directly observe such impacts. Therefore, a range of specific and verifiable short term indicators related to the aims and scope of CTMM were identified and, in addition, a survey was developed to assess the perceived impact of CTMM on longer term indicators such as accelerated translation of medical technologies to clinical use and innovation-in-society.
In line with the goals of CTMM, these process indicators were clustered to represent the scientific, translational, clinical, and economic impact of the partnership. The use of a process indicator (or marker) model has been advocated previously (Trochim et al., 2011) as a way to identify key operational and measureable markers along a generalized process pathway from research to practice. All indicators selected have previously been described in the Payback from Research Framework (Hanney and Buxton, 1996), the Research Impact Framework (Kuruvilla et al., 2006), or the translational evaluation approach (Rubio et al., 2010).
The indicators to assess scientific impact included two bibliometric indicators (i.e., the number of publications and journal impact factors, based on the Journal Citation Report (JCR) 2012 of the Thomson Institute for Scientific Information). In addition, perceived impact of CTMM on development of scientific networks was assessed.
Indicators for translational and clinical impact include number of new technologies and treatments moving from pre-clinical development to clinical testing phases; the perceived impact on acceleration of translational research in general and for biomarker and imaging technologies specifically; and the potential of clinical applications in the health care continuum.
Indicators for economic impact include usage of early health-economic modeling, which is a decision analytic approach to evaluate technologies in development so as to increase the return on investment as well as patient and societal impact (Ijzerman and Steuten, 2011). Further, the number of full time equivalents (FTEs) funded through CTMM, and the number of PhD degrees obtained, were used as a proxy for CTMM's impact on generating a highly educated work force. Also, the international background of the staff was described in order to indicate the potential of a Public–Private Partnership (PPP) like CTMM to advance the careers of scientists internationally, not only in the country wherein the PPP operates. Finally, the number of patents filed and spin-offs founded were described as a proxy for the wider economic benefits from commercial exploitation of the innovations arising from R&D.
Data collection and analysis
Data were gathered from extensive CTMM administrations, including publications, patents, project proceedings, and annual reports covering the time period 2008–2012. Perceived impact was investigated using a CTMM-wide survey. All CTMM participants, except administrative support staff, were eligible and invited by e-mail to participate in the survey and also to indicate their interest for participating in a group interview. The survey was developed in-house and face-validated by five CTMM participants prior to administration. Feedback regarding relevance and clarity of statements and answer options, as well as length of the survey, was used to design the final version. The survey included 67 closed-format items, of which 61 in the form of statements that were scored on a 5-point Likert-scale with 1 meaning very low impact/fully disagree; 5 meaning very high impact/fully agree, and 3 being a neutral option.
An exemplar statement is: “I perceive the impact of CTMM on accelerating translation of new technologies to the clinic as…,” with answering options ranging from “very low” (1) to “very high” (5). The questions were grouped in eight blocks and respondents were shown the percentage progress towards finalization of the questionnaire per block.
In addition, two group interviews with project members, principle investigators, industry representatives, and CTMM's Board of Directors were organized to reflect on and provide input on interpretation of the data collected from the administrative sources and the survey results. The group interviews were held to allow them to reflect on and provide input to the interpretation of data collected from the administrative sources and the survey results. After outlining of the goal and general process of the meeting, the results of the research were presented in three parts (scientific, clinical and translational, and economic).
We chose to use an unstructured interview format to guide the discussion of the results as this type of interviewing is recommended when the research team has developed enough of an understanding of the topic of interest to have a clear agenda for the discussion with the informants, but still remains open to having his or her understanding of the area of inquiry revised by the respondents (Warren and Karner, 2005). A note-taker was present during the interviews so the interviewer could focus on the development of rapport and dialogue. The interview notes were coded and analyzed with a view to identify actionable recommendations to the CTMM Board of Directors. The survey and group interviews were performed in 2013. All quantitative data were analysed descriptively by percentages, and means (± SD) or medians (first quartile—third quartile) depending on the data distribution. Informed consent was obtained from all participants. By Dutch law, interviewing or surveying health care professionals is exempt from medical or ethical review.
Results
Investigators were given full access to the required administrative data of CTMM. A total of 197 responses to the survey made for a small (17%), though arguably representative, sample of CTMM participants considering its public–private composition, including 25% private partner responses, 70% public partners responses (23% from universities, 47% from university medical centers and general hospitals), and 5% responses from health foundations. Further, each CTMM project was represented in the sample by both private and public partner respondents and the majority of respondents came from the cardiovascular and oncology projects (i.e., 77%). This appropriately reflects the amount of projects in these disease areas compared to the neurodegenerative, immunology, and infection, and the IT projects that make up 23% of the total projects. Twelve CTMM representatives participated in the group interviews, including 7 project leaders and 5 management team/board members of CTMM.
Scientific impact
Between 2008 and 2012, the CTMM projects collectively published 139 papers in scientific journals. The absolute majority of the papers was published by cardiovascular and oncology consortias (55 and 69, respectively), which reflects the larger number of projects in these disease areas. The average number of publications per project was 7.7 (SD = 5.4). Important to note is that, at the time of this evaluation, another 306 papers were submitted, likely resulting in a total of 445 full papers by the end of 2013.
The average impact factor of journals in which CTMM publications have appeared is consistently higher than the average JCR 2012 journal impact factor of that disease area (Fig. 3). The associated standard deviations, however, are substantial (6.8 ± 5.8 for cardiovascular; 6.1 ± 5.5 for infection and immunity; 5.7 ± 5.5 for oncology, and 4.5 ± 2.3 for neurodegenerative disease articles). This is because translational articles, such as produced within CTMM, are not necessarily published in the specific disease-oriented journals but also in general (for example, translational, multidisciplinary, or methodological) journals. Such a broad collection of journals likely sees a larger variation in their impact factors compared to disease-specific journals.
FIG. 3.
Average impact factor CTMM publications vs. average JCR (2012) journal impact factors, by disease area.
As the Dutch scientific output typically exceeds international standards in the aforementioned disease areas, the CTMM's relative output versus international standards were compared to the Dutch relative output versus the international standards. In line with a previous assessment of the Dutch scientific output versus international output, this comparison regards the combined volume and impact of journal publications (i.e., number of publications in specific journals multiplied by the impact factor of those journals). The international standard reference value is set at 100, and a value above that indicates larger output relative to the international standard. Figure 4 shows that the scientific output of CTMM is on par with the level of Dutch biomedical research output, with scientific output 15%–79% above the international reference value.
FIG. 4.
Scientific output of CTMM and Dutch biomedical research compared to International Reference Value = 100
The survey respondents perceive CTMM's impact on the volume and excellence of scientific output as high, each with a median of 4 (range: 3–4), and its impact on publishing in high impact journals as moderate (median = 3; range: 3–4). Notably, CTMM has established a strong network structure between public and private parties involved (median = 4; range: 3–4), indicated by all respondents answering affirmative to this statement. This is further illustrated by a quote from the group interviews, which was shared among the participants, that CTMM “has managed to bring together parties that otherwise would not have found effective ways to collaborate.”
Translational and clinical impact
Eleven of the 21 disease-oriented projects had progressed from pre-clinical development in 2008 to clinical testing in 2012, and another three projects were scheduled to start clinical trials in 2013. The other projects were in the phases of employing laboratory or animal experiments to develop their technologies further.
Survey respondents indicate that they perceive the impact of CTMM on accelerating translational research in general as high (median = 4; range: 3–4). The impact of CTMM on the development of biomarkers and imaging specifically is perceived by respondents as moderate to high, with somewhat larger variation between disease areas for biomarker development compared to imaging (Fig. 5a, b). In addition, CTMM is perceived to be a strong stimulus for translating new technologies to the clinic with a median of 4 (range: 3–5) across all disease areas (Fig. 5c).
FIG. 5.
(a) Perceived impact of CTMM on biomarker development by disease area; (b) Perceived impact of CTMM on imaging development by disease area; (c) Extent to which CTMM stimulates translation of new technologies to the clinic, by disease area.
By developing technologies that span more than one phase in the care continuum from prevention to treatment monitoring, CTMM aims to maximize clinical impact while exposing SME partners to a wider range of clinical environments. The majority of the CTMM projects (n = 16 of 22) develop technologies that are applicable to ≥2 of four different phases in the health care process, defined as:
• screening and prevention (n = 14): detection and assessment of risk factors;
• early diagnosis (n = 14): in vitro and in vivo detection and assessment of early disease biomarkers;
• patient stratification (n = 15): selecting the most effective therapy for the individual patient (prognostic and predictive medicine);
• image-guided treatment (n = 8): targeted drug delivery, minimally-invasive interventions, and treatment monitoring.
Economic impact
In order to guide the development of new technologies from early development stages towards solutions that will be effective as well as cost-effective in clinical practice, CTMM facilitated early health economic modelling as part of every project. Until 2012, 21 decision analytic models as well as three disease models (simulating the natural course of disease and the (likely) impact of new technology on this) have been developed.
Experience with and impact of early modeling on R&D decision-making varied widely between projects (median = 3; range: 2–4). In the project with the highest scores regarding the influence of early modeling, 50% of respondents indicated that early modeling supported R&D decision making and that, as a result of the model outcomes, changes were made in the further development of the technology (i.e., as a risk stratification instead of a screening tool) (De Graaf et al., 2012). In the project with the lowest score regarding influence of early modeling, 15% of respondents indicated impact on R&D decision making and no changes were made to technology development. Results of the group interviews indicated that, especially in the very early development stages when no concrete technology had formed yet, the integration of the health economic analyses in the projects was experienced as difficult.
One of CTMM's objectives was to generate employment growth of 300 FTE in year two. This was realized in 2010 with a capacity of 328 FTE. After that CTMM continued to grow to nearly 1900 FTE of translational R&D capacity in 2014, most of which were in academia (Fig. 6). Employee distribution by job type is 34% PhD students, 29% scientific/technical staff, 21% postdocs, and 17% senior scientists. Eleven postgraduate students achieved their PhD degree in 2012. While eleven postgraduate students achieved their PhD degree in 2012, another 246 PhD candidates are on track to graduate in 2013 to 2015, resulting in a total of 258 PhDs.
FIG. 6.
Development of Academic and Industry Full Time Equivalents (FTEs) from 2008–2014.
CTMM staff included 33 nationalities in 2012. The majority of non-Dutch staff came from Europe (Western Europe 18%, Eastern Europe 20%, and Southern-Europe 20%), followed by Asia (25%), North America (9%), Northern Europe (6%), and South America (1%).
By the end of 2012, twelve patents had been filed and four spin-off companies were founded by the CTMM members.
Discussion
Translational research moves in a bidirectional manner from one type of research to another—from basic research to patient-oriented research, to population-based research, and back—and involves collaboration between multiple public and private parties, all with their own agendas. Successfully managing a public–private partnership in translational research is a huge challenge as one has to find a way to incentivize participants, primarily for sharing expertise, ideas, data etc., and push aside the typical “publish first” or otherwise protectionist attitude that is all too common in both academia and industry. The approach to assessing the impact of translational research partnership must therefore also be flexible enough to accommodate the assessment needs of public and private partners within the consortia, but it must also be rigorous enough to document that the program is meeting its short-, intermediate-, and long-term objectives.
Against this background, an exploratory assessment of the scientific, translational, clinical, and economic impact of CTMM, as achieved 5 years after becoming operational in 2008, is described in this report. To the best of our knowledge, this is one of the few published articles across the innovation analysis literature describing the impact of a public–private partnership using the program's administrative data and additional survey information.
The results show that CTMM's scientific impact is as high as the overall impact of Dutch biomedical research (i.e., 15%–80% above international standards in terms of volume and journal impact factors). This is an important finding in light of the view held by some that translational research, particularly when carried out in public–private consortia, would be of lower quality than purely academic-driven research. In order to assess the generalizability of this finding, it is important to realize that the scientific impact of CTMM is largely driven by the quality of the academic groups involved. As such, scientific impact of a PPP is likely to be as good as the relative quality of the scientists it can attract.
Regarding translational impact, it became apparent that the majority of CTMM projects moved from bench to clinical trials within the 5-year evaluation timeframe. Notably, participants perceive that CTMM has contributed importantly to accelerating this process. While no comparison could be made with specific other PPPs, a median translation lag of 24 years reportedly exists between first description of the discovery and the earliest highly cited article (Contopoulous-Ioannidis et al., 2008). Others stated that “it takes an average of 17 years for research evidence to reach clinical practice at a rate of 50% use in the relevant population” (Balas and Boren, 2008). Compared to this, it seems that partnerships like CTMM can speed up at least the “bench to bedside”, or T1 phase of this process as described by Westfall (Westfall et al., 2007), before moving the research to practice (phase T2).
The (future) clinical impact of CTMM is large as its projects focus on disease areas with high human and economic burden. Furthermore, CTMM emphasizes the development of medical technologies that are applicable to more than one phase of the health care continuum, in order to maximize patient value and increase the probability for return on investment. To what extent the expected value of these technologies will be harvested in practice strongly depends on whether technology development and implementation continue beyond CTMM's finalization in 2015.
A particular strength of CTMM is that it has aimed to drive collaboration in research methodologies across the development cycle, notably by facilitating early health economic modeling as part of the research projects. This was done by assigning a team of health economists to each project, with the aim to explore and project potential cost-effectiveness of the technologies as an integral part of the technology development process. While the impact of such early health economic modeling on the R&D decision making has been lower than was hoped for at the start, a few champion projects (De Graaf et al., 2012; Handles et al., 2012) illustrate the potential added value of this approach.
One specific aspect to consider in order to increase the usability of early cycle economic evaluation is the choice of metrics to assess. Whereas health economists typically advocate the use of the Quality Adjusted Life Year as their primary outcome measure—as this metric can be compared across diseases and is commonly applied at later stages—other metrics may be more informative in the early stages, such as false positive and false negative rates in case of diagnostics. We suggest that future work in this area should be “strong on principles and soft on metrics,” (i.e., the metrics that resonate most clearly with the various decision makers should be used; Steuten and Ramsey, 2014).
Another barrier may have been that health economic and technology development activities ran ‘out of sync,’ for example, when modeling activities would be outpaced by rapid technology iterations or when “classical” health economic modeling was considered too data intense. Other health technology assessment approaches, including, for example, cost-effectiveness gap analyses, multi-criteria decision analyses (Ijzerman and Steuten, 2011), or minimal modeling (Meltzer et al., 2011), may be solutions in this regard.
Beyond the infrastructure created by the CTMM to integrate health economic modeling into medical technology development, there may also be a greater role for Technology Transfer Organizations to supporting early stage economic evaluations. University TTOs generally aim to facilitate, enhance, and implement the transfer of knowledge and technology created and developed within the institution towards economic value creation, by translating the results of scientific activity into formats readily adoptable and absorbable by industry and commerce. For achieving this, they deploy activities in three main “business” areas: (1) collaborative research with companies and provision of consultancy services; (2) patenting and licensing; and (3) nurturing and developing spin-off companies (Debackere, 2012). Early health economic modelling, however, is a relatively new strand of health economics that may not be commonly known among all TTOs, let alone be routinely included in TTO activities. Education about early economic modeling, and providing or connecting to consultancy services in this area, should be intensified under activity area (1) mentioned above, in order to accelerate high value new medical technologies to the market.
The economic impact of CTMM so far is mainly indicated by the nearly 1900 FTEs generated since 2008. Besides the high number of FTE's, CTMM has created a pool of young international professionals that are well-trained in multidisciplinary team-based approaches to translational research across institutions, which is an asset for the future of biomedical R&D. The long-term impact of CTMM on health care productivity, health care costs, and wider socio-economic impact has not been quantified in this retrospective assessment, but may be part of ongoing research.
As any research, this retrospective assessment has its weaknesses, such as the low response rate to the survey. Proven effective design strategies that were applied to attain good response included pilot-testing of the length of the questionnaire, personalized e-mail invitations, and showing respondents the percentage progress towards finalization of the questionnaire. Furthermore, two reminders were sent after 2 weeks and after 1 month. Future studies should budget for incentive-based strategies, as even small monetary incentives are very effective in improving response (VanGeest et al., 2007).
Also, while the production of a scientific research publication remains a good indication of research progression and knowledge generation (Allen et al., 2009), a full citation analysis would have shed more light on the contribution that individual articles made to further research (Jones et al., 2012).
More importantly, perhaps, it became apparent that the structured assessment to link the inputs and outputs of CTMM, to quantify its measurable impact and to draw lessons for improvement, was not always as informative as expected and findings were sometimes met with resistance. This is not unique to the CTMM impact assessment and similar experiences have been reported previously (Kaplan and Garrett, 2005), for example, after the assessment of the Institute for Translational Sciences, one of the CTSA sites (Scott et al., 2014). By using indicators alone, such as publications, FTEs, and even impact as perceived by participants, the current assessment frameworks do not provide the necessary depth of understanding how well a consortium such as CTMM has nurtured and supported multidisciplinary team science and translational research. In short, leadership would have appreciated more illumination of the “value added” by CTMM and sought more actionable information regarding the relevance and adequacy of their approach.
Nevertheless, the operationally-defined process indicators approach was considered useful and pragmatic, in lieu of observing longer term human, economic, and societal impact of this PPP, which would be much more resource-intense. For future evaluations, however, it is recommended to design and incorporate an impact evaluation plan from the start, identifying the most appropriate process indicators beforehand and collecting the data accordingly, instead of commissioning a somewhat ad hoc evaluation. This may also help to engage the PPP's participants more positively in the evaluation, and to achieve a deeper understanding of the critical success factors for maximizing impact. Recently, a combination of the Kellogg Logic Model and the World Health Organisation's Health Services Program Evaluation Model was proposed for this purpose, and was shown to have the desired results in the evaluation of NIH's CTSA program (Scott et al., 2014).
Conclusion
The retrospective assessment of the impact of CTMM indicates that it succeeded in bringing together parties that otherwise would not have found effective ways to collaborate and share ideas, expertise, and data. Five years after becoming operational in 2008, the CTMM has demonstrated a solid scientific impact that is on par with the overall high-quality of Dutch biomedical research. By moving more than 50% of technologies in development from bench to clinical trials in 5 years, it likely accelerated translational research. Early health economic modeling along the technology development cycle has supported R&D decision-making in some projects and provided insight in the likely cost-effectiveness of specific technologies. Overall, the (future) clinical impact of CTMM is generally perceived as large, according to academic and industrial partners alike, but could not be quantified sufficiently yet.
For future evaluations of PPPs or other institutional and organizational innovations conceived to move discovery science rapidly and effectively to clinical and societal applications, it is recommended to design and incorporate an impact evaluation plan from the start, and engage the partnership's participants in the evaluation, to achieve a deeper understanding of the critical success factors for maximizing human, economic, and societal impacts.
Abbreviations Used
- CTMM
Center for Translational Molecular Medicine
- CTSA
The Clinical and Translational Science Awards
- FTEs
Full Time Equivalents
- IT
Information Technology
- JCR
Journal Citation Report
- NIH
The National Institutes of Health
- R&D
Research and Development
- SMEs
Small and Medium Enterprises
Acknowledgments
I thank the project members, principal investigators, industry representatives, and CTMM's Board of Directors who participated in the group interviews. The collection and analyses of the data and the preparation and reporting on the group interviews would not have been possible without Martijn Schutten. Maarten IJzerman is acknowledged for providing input to an initial draft of this manuscript. Many thanks go to Annika Hanson for proofreading, editing, and overall assistance in preparing the manuscript for submission.
Author contributions. LS has 1) made substantial contributions to conception and design, acquisition of data, analysis and interpretation of data; 2) drafted the manuscript and revised it critically for important intellectual content; 3) given final approval of the version to be published; and 4) agrees to be accountable for all aspects of the work.
Author Disclosure Statement
Dr. Steuten reports stock ownership of PANAXEA b.v., The Netherlands, which received funding from the Center for Translational Molecular Medicine for performing this assessment. The funders did not have an influence on the design, analysis, and/or reporting of the manuscript contents. The views and analyses expressed in this article are solely the author's independent personal opinions, and do not necessarily reflect the positions of the author's affiliated organizations nor the funders.
References
- Allen L, Jones C, Dolby K, Lynn D, and Walport M. (2009). Looking for landmarks: The role of expert review and bibliometric analysis in evaluating scientific publication outputs. PLoS One 4, e5910. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Balas EA, and Boren SA. (2008). Managing clinical knowledge for healthcare improvement. Yearbook of Medical Informatics 2000. Stuttgart: Schattauer Verlagsgeselschaft mBH, 321, 1298–1299 [PubMed] [Google Scholar]
- Chakma J, Sun GH, Steinberg JD, Sammut SM, and Jagsi R. (2014). Asia's ascent—Global trends in biomedical Rand D expenditures. NEJM 317, 3–6 [DOI] [PubMed] [Google Scholar]
- Contopoulous-Ioannidis DG, Alexiou GA, Gouvias TC, and Ioannidis JP. (2008). Life cycle of translation research for medical interventions. Science 321, 2. [DOI] [PubMed] [Google Scholar]
- De Graaf G, Postmus D, Hillege H, Steyerberg EW, and Buskens E. (2012). A method for the early health technology assessment of novel biomarker measurement in primary prevention programs. Stat Med, 31, 2733–2744 [DOI] [PubMed] [Google Scholar]
- Debackere K. (2012). The TTO, a university engine transforming sciences into innovation. League of European Research Universities 10. Pages 1–16
- Handles RL, Aalten P, Wolfs CA, et al. (2012). Diagnostic and economic evaluation of new biomarkers for Alzheimer's disease: The research protocol of a prospective cohort study. BMC Neurol 12, 72. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Hanney S, and Buxton M. (1996). How can payback from health services research be assessed? J Health Serv Res Policy 1, 8. [PubMed] [Google Scholar]
- Ijzerman MJ, and Steuten LM. (2011). Early assessment of medical technologies to inform product development and market access: A review of methods and applications. Appl Health Econ Health Policy 9, 331–347 [DOI] [PubMed] [Google Scholar]
- Jones TH, Donovan C, and Hanney S. (2012). Tracing the wider impacts of biomedical research: A literature search to develop a novel citation categorisation technique. Scientometrics, 93, 125–134 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Kaplan SA, and Garrett KE. (2005). The use of logic models by community-based initiatives. Eval Program Plan 28, 167–172 [Google Scholar]
- Kuruvilla S, Pleasant A, and Walt G. (2006). Describing the impact of health research: A Research Impact Framework. BMC Health Serv Res 6, 17. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Meltzer DO, Hoomans T, Chung JW, and Basu A. (2011). AHRQ Methods for Effective Health Care. Minimal Modeling Approaches to Value of Information Analysis for Health Research. Rockville (MD): Agency for Healthcare Research and Quality (US) [PubMed] [Google Scholar]
- Rubio DM, Schoenbaum EE, Lee LS, et al. (2010). Defining translational research: Implications for training. Acad Med 85, 470–475 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Scott CS, Nagasawa PR, Abernethy NF, et al. (2014). Expanding assessments of translational research programs: Supplementing metrics with value judgements. Eval Health Prof 37, 83–97 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Steuten LM, and Ramsey SD. (2014). Improving early cycle economic evaluation of diagnostic technologies. Expert Rev Pharmacoecon Outcomes Res 14, 491–498 [DOI] [PubMed] [Google Scholar]
- Trochim W, Kan C, Graham M, and Pincus HA. (2011). Evaluating translational research: A process marker model. Clin Transl Sci 4, 153–162 [DOI] [PMC free article] [PubMed] [Google Scholar]
- Vangeest JB, Johnson TP, and Welch VL. (2007). Methodologies for improving response rates in surveys of physicians: A systematic review. Eval Health Prof 30, 303–321 [DOI] [PubMed] [Google Scholar]
- Warren C, and Karner T. (2005). Discovering Qualitative Methods: Field Research, Interviews and Analysis. Los Angeles, CA, Roxbury Publishing Company [Google Scholar]
- Westfall JM, Mold J, and Fagnan L. (2007). Practice-based research—“Blue Highways” on the NIH roadmap. JAMA 297, 3. [DOI] [PubMed] [Google Scholar]





