Methods for Identifying Translational Researchers

Abstract

There is currently no generally accepted method for identifying the community of translational researchers when evaluating Clinical and Translational Science Centers. We use data from the multiyear evaluation of the University of Illinois at Chicago Center for Clinical and Translational Science (CCTS) to investigate the complexities of reliably identifying translational researchers. We use three methods to identify translational researchers: (1) participating in CCTS services and programs; (2) self-identifying as a translational researcher; and (3) engaging in activities that are characteristic of translational science. We find little overlap of these differently defined research groups. We conclude with a discussion of how the findings suggest challenges for evaluating translational science programs and the need for better definition, communication, and demonstration of translational science for scientists and evaluators.

Introduction

Recognizing that medical science could be more efficiently aligned with society's needs, the National Institutes of Health (NIH) established a major initiative in 2006 to develop programs for clinical and translational science (Woolf, 2008). NIH defines two components of translational research as (1) “the process of applying discoveries generated during research in the laboratory, and in preclinical studies, to the development of trials and studies in humans” and (2) “research aimed at enhancing the adoption of best practices in the community”¹ (Request for Information, 2010, ¶ 4). At this writing, 61 academic institutions are funded through the Clinical and Translational Science Awards (CTSA) Program of NIH to form a Consortium intended to (1) create an academic home for clinical and translational research and (2) increase the capacity to conduct this research at the local, regional, and national level. While each CTSA-supported center has its own aims and approach, all are committed to developing services for researchers to conduct clinical and translational research. The contributions of the CTSA Program were reaffirmed in a 2013 Institute of Medicine review (Leschner, Terry, Schult, & Liverman, 2013).

Evaluation is an important component of the CTSA initiative, with each center responsible for assessing impacts and progress. To do so, evaluators must identify metrics that capture translational activities, outputs, and outcomes. Distinguishing translational from nontranslational researchers is a challenging process that often relies on the scientist. There is no currently standard, generally accepted method for identifying translational researchers when evaluating CTSAs. As a result, several important research questions arise: (1) Do scientists consistently self-identify as translational researchers? (2) What activities do translational researchers report doing? (3) To what extent is there overlap in scientist perspectives with the receipt of CTSA services? In addressing these three questions, we will compare three ways in which evaluators can operationally identify translational researchers: (1) participation in one or more Center for Clinical and Translational Science (CCTS) services, (2) scientist self-identification, and (3) engaging in translational activities, exploring the relationships, and overlap among these methods.

Methods and Analysis

This research draws from the evaluation of the University of Illinois at Chicago (UIC) CCTS which began operations in 2007 and received its first NIH CTSA in 2009.

We took a multipronged approach to identifying translational researchers in the UIC CCTS community, which includes faculty, researchers, staff, and local community members. Our analysis is based on two data sources, the evaluation database used to track all CCTS participants and two Annual Scientific Collaboration Surveys (ASCS). The following subsections present the three identification metrics and overlap among them. A final analysis section provides logistic regression results showing similarities and differences among explanatory variables regressed on the identification metrics.

Identifying Translational Researchers Based on CCTS Participation

Because the CTSAs are designed to provide the resources and support that is critical to clinical and translational research (Zerhouni & Alving, 2006), it is reasonable to expect that most individuals who interact with the CTSA supported UIC CCTS have the potential to advance translational activities. We define CCTS participants as those who participated in any of the following services or activities provided by the UIC CCTS: pilot grant funding, training, assistance on research proposals and institutional review board applications, data analysis assistance, course instruction and training, and attendance at lectures or seminars. While some evaluation teams do not include attendance at lectures and seminars as an indicator of CTSA participation, for our purposes, we wanted to be as inclusive as possible. We track all individuals who use CCTS services provided by seven cores: Design and Analysis, Clinical Interface, Biomedical Informatics, Regulatory Support, Advocacy and Bioethics, Community Engagement, and Research Education and Careers in Health. Each core maintains records of each service occurrence including dates, names, and university identification numbers. These data are periodically updated and merged into a relational database that links all service use data with other individual-level data, enabling identification of researchers, faculty, students, and community members who might be engaged in translational activities. In 2009, there were 356 individuals who received a CCTS service or participated in sponsored activities, 926 in 2010, and 958 in 2011.

Self-Identification of Survey Respondents as Translational Researchers

Our second method for identifying translational researchers is to ask individuals if they conduct translational research. The benefit of this approach is that it does not assume that use of CCTS services is indicative of doing translational research. Moreover, we are able to see if researchers describe their work as translational. The drawback of this approach is that it is possible that researchers are engaged in translational research, but do not define their work with such terms. It is conceivable that an individual might describe her research as clinical research with human subjects and health services research, and although she also engages in translational work, she does not define herself with such terminology, or views the translational component as too minor to report.

We collect self-identification information as part of the ASCS, which is administered to a sample of CCTS participants and a random selection of nonparticipant faculty from the university's seven health-related colleges. In 2010, the survey was administered to 938 CCTS participants and 499 nonparticipants. The response rate was 39.2%, including 415 CCTS participants (44.2%) and 149 nonparticipants (29.9%). The 2011 survey was administered to 1,538 CCTS participants and a random sample of 1,049 nonparticipants; the overall response rate was 35.3%, and included 590 participants (38.4%) and 325 nonparticipants (30.1%).

We asked two survey items, preceded by definitions of translational research consistent with that employed by NIH: “(1) The process of applying discoveries generated during research in the laboratory, and in preclinical studies, to the development of trials and studies in humans; and (2) Research aimed at enhancing the adoption of best practices in the community.” The first item provided the following list from which respondents were asked to identify all of the types of research they conduct: (1) clinical research with human subjects, (2) laboratory-based research with human specimens, (3) laboratory-based research without human specimens, (4) population-based, epidemiological or public health research, (5) translational research, (6) educational research, (7) health services research, (8) practice-based research, (9) community-based research, (10) engineering research, (11) other (please specify), and (12) not applicable. The question asked “What type(s) of research do you do?” This list of 12 items was developed based on knowledge gained from interviews of researchers and project leaders conducted as part of the evaluation. Self-identification occurs when individuals select the translational research option. Because respondents were allowed to select multiple types of research, we expect that those who perceive their work as translational would have selfidentified. We call this measure Undirected Self-Identification.

The second item asked more specifically: “During the past academic year (August 2009—August 2010), did your work involve translational science or translational research?” enabling identification of respondents who might not describe their research as translational, but remain involved in translational research activities through collaborations, in the classroom, or through service activities. We call this measure Directed Self-Identification.

Table 1 presents frequencies for these questionnaire items. In 2010, 138 (27.1%) of those who completed the undirected item (n = 510) self-identify as doing translational research. In comparison, 265 (52.7%) affirm that their work in the past year involved translational science or research (n = 503). Table 1 indicates that in 2010 more individuals report being involved in translational science or research (directed) than define their work as translational (undirected). Thus, self-identification, and hence the foundation of evaluative efforts, is clearly related to the design of the survey question used. It is possible that researchers do not describe their overall research agenda to be translational, but they do consider some of their research to be translational. Additionally, the proportions of those reporting involvement in translational science and research in 2011 are relatively consistent with the proportions reported in 2010, indicating that respondents consistently interpret the directed self-identification question.

Table 1.

Frequency of Measures.

Frequency of translational science indicators		Count	%
What type/types of research do you do? Translational research, 2010 (undirected self-identification)	No	372	72.9
	Yes	138	27.1
	Total	510	100.0
During the past academic year (August 2009–August 2010), did your work involve translational science or translational research? (directed self-identification)	No	238	47.3
	Yes	265	52.7
	Total	503	100.0
During the past academic year (August 2010–August 2011), did your work involve translational science or translational research? (directed self-identification)	No	324	47.2
	Yes	362	52.8
	Total	686	100.0
Frequency for translational science indicators, 2010, N = 565		Yes	%
CCTS participant		415	73.5
Conducted at least one translational activity		236	41.8
One translational self-report + I Activity		150	26.5
Two translational self-report		120	21.2
Two translational self-reports + Participant		96	17.0
Two translational self-reports + I Activity		59	10.4
Two translational self-reports + Participant + I Activity		47	8.3

Note. CCTS = Center for Clinical and Translational Science.

One of the advantages of asking translational researchers to self-identify is that we can investigate whether or not these researchers are connecting with the campus CCTS, whose primary goal is outreach and service to translational researchers. In 2010, more individuals affirm the directed question (248) than the undirected question (129). Among the 248 who did work that involved translational science (directed self-identification), 197 (79%) were participants and 51 (21%) were not. Of the 345 CCTS participants, more than half (197) reported doing work that involved translational research. Of those who reported doing translational research (undirected self-identification), 107 (83%) were CCTS participants and 22 (17%) were nonparticipants. A cross-tabulation shows that in 2010 there was a significant difference in doing translational work across CCTS participants (352) and nonparticipants (119; χ ² = 6.3, df = 1, p = .01).

This comparison enables us to identify a group of translational researchers who have not used CCTS services and also indicates that the CCTS might be providing services and opportunities for people who do not self-identify as translational researchers (directed or undirected). For example, 51 individuals who affirmed the directed self-identification item reported no affiliation with the campus CCTS. Among individuals who affirmed the undirected question in 2010, CCTS participants are more likely to self-identify (χ ² = 7.2, df = 1, p = .007). Additionally, a substantial proportion of individuals not involved in translational research are using CCTS services (43% of CCTS participants do not report doing translational research).

Comparing the reports on CCTS participation and self-identification of translational researchers demonstrates fundamental challenges for evaluating CCTS progress and outcomes. The findings demonstrate the generally low to moderate coincidence across measurement methods for identifying translational scientists. While evaluators must select an identification metric, the choice has implications for determining program outputs and outcomes.

The ASCS enables us to assess change in responses to the directed question asking respondents if their past years work involved translational science or research. Among the 237 individuals who responded to the item in both 2010 and 2011, 96 (40.5%) responded in the affirmative both years, while 31 (13.1%) responded “yes” in 2010 and “no” in 2011, and 28 (11.8%) responded “no” in Year 1, but “yes” in Year 2. Three interpretations of these findings are possible. First, the findings may indicate learning effects that occurred after the establishment of the CCTS such that respondents became better able to assess whether they actually conducted translational work. Second, the question may provide too much room for respondent interpretation resulting in inaccuracy. Another possibility is that answers will vary across years depending on whether researchers have translational grants or the type of work that is underway on their projects.

In addition to assessing responses about doing translational research, we are interested in the intersection of self-identification and CCTS participation. We also compared the distribution of CCTS participants and nonparticipants who responded affirmatively to the directed survey question in 2010 and 2011. In both years, among those who affirmed that their work involved translational research, the majority participated in CCTS services or activities (78% in 2010 and 70% in 2011). Among participants, 59% reported being involved in translational research in 2011, up from 55% in 2010. In both 2010 (χ² = 7.2, df = 1, p = .007) and 2011 (χ² = 7.2, df =1, p = .000), there is a significant difference between participants and nonparticipants who responded positively to the directed survey question. Of particular interest are the individuals who are not using CCTS services, but report doing work that involved translational research (51 in 2010 and 107 in 2011), representing an untapped audience for the campus CCTS.

Engaging in Activities That Characterize Translational Research

The third approach used to identify translational researchers was to ask respondents whether they had undertaken any of the eight activities aimed at communicating research findings to nonacademic communities. Some of the activities were drawn from a planned 2010 Dutch study that evaluated translational science outcomes in the Netherlands and were provided by science policy researchers at the Rathnau Institute. Others were developed by the authors and reflect knowledge gained from interviews. The ASCS asked “During the past academic year (August 2009–August 2010), have you… [e.g. ‘Contributed to a media report’].”

Table 2 outlines the frequency of each translational activity reported by respondents in 2010 and 2011. In 2010, the most common activity was “presenting to a non-scientific audience” (39%), followed by “serving on a committee that is developing guidelines or policy recommendations” (25%), and “contributing to a media report” (24%). This pattern holds for 2011. The least common activity reported in both years was “teaching a course for policy makers or professionals.”

Table 2.

Frequency of Translational Activities, 2010 and 2011.

During the past academic year, have you…	2010			2011
	Yes	Total	%	Yes	Total	%	% Change
Contributed to a media report	89	370	24	124	508	24	0.4
Published in a journal that is directed to policy makers or practitioners	86	371	23	125	504	25	1.6
Contributed to a policy report	59	370	16	94	503	19	2.7
Presented to a nonscientific audience	144	372	39	206	506	41	2.0
Taught a course for policy makers or professionals	41	369	11	52	494	11	‒0.6
Served on a committee that is developing guidelines or policy recommendations	91	370	25	133	502	26	1.9
Served on a review committee that awards funding for clinical or translational (bio)medical and health research	59	370	16	104	501	21	4.8
Served as an editor for (bio)medical or health research journals that target professionals and practitioners	49	369	13	91	499	18	5.0

While this approach has the benefit of anchoring responses in actual activities, the list of activities is likely not comprehensive. However, when we offered respondents the opportunity to report translational activities in an open-ended item, they overwhelmingly specified activities that involved disseminating information, developing best practices, and implementing programs in the community (the open-ended responses are available on request). Thus, we expect that the items specified capture the major types of activities that qualify as translational.

Comparing the Three Methods

Table 3 shows a cross-tabulation between types of translational activities conducted and whether or not the respondent answered affirmatively to the directed self-identification question. Having done translational work in the previous year is significantly related to conducting seven of the eight activities in 2010 and all eight activities in 2011.

Table 3.

Cross-tabulation of Translational Activities by Whether or Not Respondent Did Translational Science or Research (Directed Item) in the Past Academic Year, 2010 and 2011.

		2010			2011
		Did Translational Work			Did Translational Work
		Yes	No	Pearson χ2	Yes	No	Pearson χ2
Contributed to a policy report	Yes	42	17		9.62***	63		30	10.64***
	No	152	157		198	206
Published in a journal that is directed to policy makers or practitioners	Yes	56	29	7.84**	86	38	19.91***
	No	138	146		173	201
Contributed to a media report	Yes	62	26	14.16***	82	41	13.68***
	No	133	147		180	199
Presented to a nonscientific audience	Yes	85	57	4.74*	130	75	17.51***
	No	110	118		131	164
Taught a course for policy makers or professionals	Yes	24	170	0.60	38	13	12.14***
	No	17	156		213	224
Served on a committee that is developing guidelines or policy recommendations	Yes	61	30	9.94***	90	42	18.14***
	No	133	144		169	194
Served on a review committee that awards funding for clinical or translational (bio)medical and health research	Yes	41	18	7.93**	70	34	12.17***
	No	153	156		188	203
Served as an editor for (bio)medical or health research journals that target professionals and practitioners	Yes	32	17	3.67*	62	29	11.17***
	No	161	157		196	206

Note. Exact significance (one-sided)

^*p < .05;

^**p < .01;

^***p < .001.

Notably, in both 2010 and 2011, the majority of researchers who selfidentified as doing translational work did not engage in any of the translational activities listed in Table 2, indicating a disconnect between what researchers perceive to be translational research when they self-identify and the practical activities associated with translating research to broader, multidisciplinary scientific communities. This finding indicates a possible functional distinction between the two components of translational research as defined by NIH: (1) the process of applying research discoveries to preclinical studies and trials and (2) the translational process from research to practice in the broader health care community. Our results suggest that the activities do not fully capture translational research as characterized by the NIH definition or that there is misunderstanding about what constitutes translational science. Evaluators, and perhaps translational scientists, need to identify a valid and representative set of activities recognized by scientists to be translational. Similarly, perhaps, there should be better communication among medical scientists about what is translational.

Figure 1 illustrates the coincidence among the three approaches, showing the overlap over time: (1) CCTS participants and nonparticipants, (2) those whose work involved translational research in the past year (directed self-identification), and (3) those who reported doing at least one translational activity. Figure 1 shows that in 2010, 30% of individuals are captured by all three approaches, 39% fall in two of the groupings, and 27% are captured by only one measure and in 2011, 24% of the individuals are captured by all three approaches, 37% by two categories, and 29% by one approach.

Figure 1.

Venn diagrams of CCTS participants, work involved translational science, and translational activities, 2010 and 2011. CCTS = Center for Clinical and Translational Science.

Figure 1 illustrates the importance of evaluating CCTS activities among participants and a random sample of nonparticipants. By capturing this comparison group, we are able to identify researchers on campus who are engaged in translational science activities, but not affiliated with the CCTS or utilizing its resources. For example, Figure 1 indicates that 65 (19%) respondents in 2010 and 147 (32%) respondents in 2011 did not use CCTS services but conducted translational science or were engaged in translational activities.

Further Comparison: Correlates of Combined Indicators of Translational Science

Finally, we sought to investigate the overlap of activities and whether or not certain types of researchers are more likely to be captured using various combinations of our multiple measures of translational research as compared to others. We created a set of dependent variables based on the previously discussed items that indicate (1) CCTS participant or not, (2) self-report of engaging in translational research (undirected); (3) selfreport of having done translational research in the previous year (directed); and (4) conducting at least one of the eight translational activities listed in the questionnaire. We created five new combined variables:

1. Affirmation on one self-report question and report of at least one translational activity.

2. Affirmation on both self-report survey questions.

3. Affirmation on both self-report questions and CCTS participation.

4. Affirmation on both self-report questions and report of at least one translational activity.

5. Affirmation on both self-report questions, at least one translational activity, and CCTS participation (all 4 items).

Table 1 notes the descriptive statistics for the original questionnaire items and the five new variables.

Table 4 shows the results of logistic regression models predicting the likelihood that respondents are captured by one or more of the questionnaire items for translational science and activities. Model 1 indicates age is positively related to self-reporting doing translational science (director or undirected) and reporting at least one translational activity and that tenure-track and research faculty are both more likely to self-report translational research and doing at least one activity. Models 2–4 also indicate that tenure-track faculty, as compared to clinical, research, and nontenuretrack faculty, and nonfaculty are more likely to respond positively to the two self-reported measures of conducting translational research; two selfreports of translational research and being a recipient of CCTS services (Model 3); and that they do translational research (as indicated by two self-report measures) and conduct at least one translational activity (Model 4). Models 2–4 also show that research faculty are significantly more likely than others in the sample to report this combination of activities. The results in Table 4, taken together, show that the primary predictor of each measure (and combination of measures) for translational research and activities is job title, or position. It is likely that tenure-track and research faculty are better positioned to engage in translational science and activities, or their positions might require that they participate in these types of activities.

Table 4.

Logistic Regression Models Predicting Multiple Measures of Translational Research.

	(1) One Self- Report of Translational Research + 1 Activity				(2) Two Self-Reports of Translational Research				(3) Two Translational Self-Reports + Participants				(4) Two Translational Self-Reports + 1 Activity
	B	SE	Exp(B)	Significance	B	SE	Exp(B)	Significance	B	SE	Exp(B)	Significance	B	SE	Exp(B)	Significance
Female	.646	.342	1.908	+.059	–.396	.428	.673		–.550	.490	.577		.084	.492	1.088
White	.236	.382	1.267		–.711	.458	.491		–.453	.504	.636		–.572	.555	.564
Age	.019	.017	1.019		–.038	.024	.963		–.038	.028	.962		–.029	.029	.972
Assistant professor	–.530	.525	.589		.220	.664	1.246		–.069	.749	.933		–.453	.794	.636
Associate Professor	–.675	.641	.509		.168	.798	1.183		.567	.850	1.763		–.901	1.055	.406
Full Professor	–.679	.673	.507		–.389	.886	.678		–.471	1.058	.624		–.683	.983	.505
Tenure track	1.866	.444	6.463	.000***	2.541	.614	12.692	.000***	1.768	.698	5.859	.011*	4.149	1.132	63.377	.000***
Clinical track	.230	.547	1.258		.078	.943	1.081		.430	.998	1.538		2.008	1.343	7.446
Research track	1.916	.577	6.791	.001**	1.548	.771	4.700	.045*	1.667	.861	5.298	+.053	3.218	1.234	24.985	.009**
Constant	–3.193	.966	.041		–.259	1.208	.772		–.355	1.334	.701		–2.807	1.700	.060
χ2	36.825				36.084				14.322				35.777
Significance	.000				.000				.111				.000
–2 Log likelihood	226.868				152.786				126.334				117.278
Cox and Snell R2	.162				.159				.066				.157
Nagelkerke R2	.225				.267				.135				.303

Note. Included in analysis = 209;

Total n = 618.

^***p < .000;

^**p < .01;

^*p < .05;

⁺p < .10.

Overall, these results point to important implications for future evaluations of university translational research and activities. First, the results indicate a distinction between the two components of translational research: (1) the translation from research laboratory to clinical application and trials and (2) the translation of research to communities of practice. Individuals who self-report as having done translational research in the previous year are not necessarily the same individuals who report translating research to practice. Specifically, among those who reported doing translational research (265 in 2010 and 362 in 2011), only a fraction reported contributing to media reports (31.8% in 2010; 31.3% in 2011) and policy reports (21.6% in 2010; 24.1% in 2011); developing guidelines or policy recommendations (31.4% in 2010; 34.7% in 2011); or teaching a course to policy makers and professionals (12.4% in 2010; 15.1% in 2011). Defining translational researchers and scientists by policy work and practice may result in different categorizations of individuals as compared to self-identification and CTSA affiliation. How evaluators define and identify translational researchers can result in distinct concepts and measures, indicating the importance of thinking critically about the ways in which we define and categorize researchers in CTSA evaluation activities.

Conclusion

This analysis compared three different methods for identifying translational researchers, a key first step to a valid evaluation methodology that relies on the individual as a unit of analysis. Three research questions were addressed. We found that scientists are moderately consistent in selfidentifying, or not, as translational researchers (Research question 1). In addition, translational researchers were more likely to report contributing to policy reports, publishing in journals that are targeted toward policy makers, contributing to media reports, presenting to nonscientific audiences, serving on committees that develop guidelines or policy recommendations, serving on review committees that award funding for clinical or translational research, and serving as an editor for a medical or health journal (Research question 2). We also found that most researchers who reported doing translational research also participated in CCTS services (Research question 3).

However, when the three different methods applied here—participation, self-identification, and activity—are examined together, our findings indicate only low to moderate overlap, and that even within methods, the formulation of the metric determines the community captured. The findings allow us to make four tentative conclusions: (1) the inherent fuzziness of the current conceptualization of translational research; (2) strategies for evaluation, given the fuzziness; (3) opportunities for CTSAs to use alternative methods to assist with management and capacity development; and (4) needs for the medical science community to provide better definitional guidance about what constitutes translational research.

The fuzziness of the current conceptualization of translational research likely stems from several sources. Scientists have an a priori mental image of what constitutes translational research that is likely broader than the NIH definition. Additionally, the UIC CCTS appears not to turn away scientists who are not translational researchers. The confusion about what is and is not translational is likely increased by the expansive a priori conceptualization of translational science and the openness of the UIC CCTS to provide services.

Evaluators are tasked with assessing the impact of CTSA services on translational science in their universities. Given the findings, two recommendations are clear. First, evaluators must consider multiple methods of identifying translational researchers (and likely translational outcomes). Second, there should be continuous measurement of all methods and metrics over time. Comparing participation in one year with self-identification or activity in another year is not appropriate. Rather, evaluators need to select and track a small set of indicators over time, and to include both service use tracking and survey methods. Additionally, evaluators should better instrument their measures of translational science such that they capture both dimensions of the NIH definition—lab to bedside and research to community of practice. The questions used to capture translational activities in this article relate primarily with the second dimension.

Several limitations of this research should be considered. Perhaps most importantly, the optimal strategy for defining and identifying the universe of potential translational researchers at any institution is itself unclear. Whether this would best include all faculty in biomedical disciplines or programs, all researchers with approved human subjects protocols, or all researchers who have applied for NIH support is not obvious. Of course, the mobility of researchers across institutions makes definitive identification of the translational research community at any university even more problematic. Developing adequate operational definitions of scientist's affiliation with translational research and translational science research remain basic questions that warrant further investigation.

The different methods and approaches examined here provide alternative windows into the groups who use services and the overlap of activities and perceptions. In large initiatives such as the CTSA Program, it is not unusual for there to be continual learning and evolution in thinking before consensus is reached. These insights could be used to market services and better define the context and components of translational research. At the national level, such efforts could better articulate translational science policy as a frame for individual CTSA programs and enable NIH to better understand the ways in which researchers describe and identify their own research as translational or not.