Abstract
A conceptual model was developed to guide evaluation of the long-term impacts of research grant programs at the National Institutes of Health, National Institute of Environmental Health Sciences. The model was then applied to the extramural asthma research portfolio in two stages: (1) the first used extant data sources, (2) the second involved primary data collection with asthma researchers and individuals in positions to use asthma research in development of programs, policies, and practices. Reporting on the second stage, this article describes how we sought to broaden the perspectives included in the assessment and obtain a more nuanced picture of research impacts by engaging those involved in conducting or using the research.
Program evaluation is critical to understanding and documenting the impact of research programs. Yet there are few comprehensive models for measuring long-term research impact suitable for use by health and environmental research organizations such as the National Institute of Environmental Health Sciences (NIEHS) of the National Institutes of Health (NIH) (Engel-Cox et al., 2008; Coryn et al., 2007; Trochim et al., 2008; Van Houten et al., 2000). This is especially the case for research grant programs because, by definition, grants have indirect benefit to and little substantial involvement by federal agencies (Engel-Cox et al., 2008).
Most empirical studies address the link between research inputs and outputs, with bibliometric analysis as the dominant methodology. Methods to link research outputs (primarily publications) to short-term research outcomes are somewhat more varied and include citation analyses, surveys, publication analyses, and key informant interviews with policy makers. Comparatively few studies have assessed intermediate or long-term outcomes (Powers et al., 2006). A few notable attempts have been made to develop impact models for the research context. Hanney and colleagues (Buxton and Hanney, 1996; Hanney et al., 2003) present an input-output model for assessing “payback” from applied research. Rubenstein and Geisler (1988) use a conceptual model of the linkages between the research and development process and social systems. Bozeman (2003) uses a public value mapping model for research evaluation. Kuruvilla et al. (2006) developed a research impact framework to guide analysis of the impact of selected research projects at the Long School of Hygiene and Tropical Medicine. Trochim and colleagues (2008) developed a logic model encompassing short-term, intermediate, and long-term outcome markers to drive a mixed-method evaluation of a large National Cancer Institute research initiative.
The NIEHS sponsored a series of interrelated studies from 2005 to 2008 designed to develop and test a conceptual model of its extramural research programs (Engel-Cox et al., 2008). We explored the feasibility of designing research impact assessments to test this model through application to the NIEHS extramural asthma research program using both archival (Liebow et al., 2009) and primary data (present study).
NIEHS asthma research portfolio
The mission of NIEHS is to reduce the burden of human illness and dysfunction from environmental causes. This mission is carried out, in part, through extramural research programs that address diseases for which there is a strong indication of an environmental component and high, or increasing, prevalence in the U.S. population (NIEHS, 2006). For example, NIEHS supports a multifaceted asthma research program, including (1) basic research into asthma genetics, respiratory pathogenesis and mechanisms by which particulate matter, aerosols, and aeroallergens induce asthmatic response; (2) new methods to measure exposure to dust and airborne allergens associated with asthma; (3) national surveys to assess the risks of exposure to allergens and other environmental agents associated with asthma; (4) the effectiveness of interventions that lower allergen levels in the home; and (5) the effectiveness of interventions for secondary prevention of asthma. Basic research is funded mainly through investigator-initiated research grants. The exposure, risk assessment, and intervention studies are supported through translational research mechanisms funded by the NIEHS Division of Extramural Research and Training. The overall goal of the program is to improve the scientific basis for reducing the morbidity, mortality, and other public health effects of asthma, which have progressively increased over the last 15 years and which are disproportionately distributed among children, minorities, and persons of lower socioeconomic status.
Evaluation approach
A conceptual model was developed to guide NIEHS research impact assessments. The model reproduced in Figure 1 postulates pathways to link NIEHS-funded research to ultimate outcomes (Engel-Cox et al., 2008). This model was applied to the NIEHS extramural asthma program in two stages. The first stage explored the extent to which extant data sources could assess asthma research funding impacts on science, clinical policy and practice, environmental policy and practice, and health outcomes (Liebow et al., 2009). This effort demonstrated that it is possible to conceptualize program components, mine existing databases, and begin to show program impacts using existing data. However, to create a more complete picture of portfolio impacts, primary data collection with asthma researchers, clinicians, science policy experts, public health advocates, and other stakeholders was recommended.
The second stage (present study) explored research impact through primary data collection with asthma researchers and individuals in positions to use asthma research in development of policies and practices. By engaging those involved in conducting or using the research, we sought to broaden the perspectives included in the assessment and obtain a more nuanced picture of research impacts.
Methods
We used two primary data collection methods: (1) a survey of individuals who have received asthma research funding from NIEHS or comparison federal agencies and (2) key informant interviews with end users of asthma research. By design, individuals surveyed (asthma researchers) are positioned on the left-hand side of the model (e.g., inputs and activities). The survey explored how respondents’ body of research had affected model elements moving to the right from activities to outputs and then to outcomes. Survey topics included asthma research funding, research dissemination, product development and commercialization, research impacts, and research gaps. The end user interviews, in contrast, were conducted with individuals associated with the right-hand side of the model (e.g., outcomes). Through semi-structured telephone interviews, we asked interviewees to reflect on how they access information generated through research (left-hand side of the model), how this information is used in their work, and how it contributes to outcomes. Interview guides addressed awareness and use of research, outcomes associated with their work, and recommendations for future research and translation. The survey and interview guides were approved by the Battelle Institutional Review Board; the survey was approved by the Office of Management and Budget (No. 0925-0588).
Survey results
The researcher survey was directed to all investigators who received any asthma research funding from the selected NIH agencies shown in Table 1 or the Environmental Protection Agency (EPA) during the period 1975 through 2005, a 30-year time frame chosen to increase the likelihood that outcomes in the model could occur. Eligible respondents were identified through an NIH-wide database of extramural research and training grants (IMPAC II) and through the EPA STAR grants database. A total of 725 asthma researchers completed the web-based survey during a three-month period in 2008 for an overall response rate of 63 percent. Most respondents had a PhD (57%), an MD (48%), or both (8%) and had received their highest degree since 1970 (88%). Non-responders were more likely to have never received NIEHS funding and to have last received asthma-related funding prior to 1980.
Table 1.
Background of Respondents
| Percentage of Respondents | ||
|---|---|---|
| Asthma Research Funding* | Ever Received | Primary** Last 10 years |
| NHLBI | 65.3 | 41.8 |
| NIAID | 39.7 | 19.7 |
| NIEHS | 31.3 | 12.0 |
| NICHD | 4.3 | 0.7 |
| Other NIH | 15.2 | 1.6 |
| EPA | 18.4 | 4.2 |
| CDC | 12.9 | 4.4 |
| AHRQ | 8.1 | 3.1 |
| Foundations | 49.6 | 0.2 |
| Industry | 41.3 | 0.3 |
| University discretionary/ start-up funds | 45.7 | 4.0 |
| Local, state or regional government | 23.3 | 3.4 |
| Other agency (e.g., NSF, HUD, FDA) | 8.8 | 1.5 |
NHLBI = National Heart, Lung and Blood Institute, NIAID = National Institute of Allergy and Infectious Diseases, NIEHS = National Institute of Environmental Health Sciences, NICHD = National Institute of Child Health and Human Development, EPA = Environmental Protection Agency, CDC = Centers for Disease Control and Prevention, AHRQ = Agency for Healthcare Research and Quality, NSF = National Science Foundation, HUD = Department of Housing and Urban Development, FDA = Food and Drug Administration
Primary refers to the Institute/Agency that provided the highest funding (dollar value) to each respondent in the previous 10 years.
Inputs and activities
For this group of researchers, the primary funding source for asthma-related research in the past 10 years was NIH, with the National Heart, Lung and Blood Institute (NHLBI), the National Institute of Allergy and Infectious Diseases (NIAID), and NIEHS topping the list of specific agencies. Despite the dominance of NIH as a primary funding source (74%), other funding sources were tapped, including foundations, industry, university discretionary funding, other federal agencies, and local, state, or regional government (Table 1).
Federal funding was dominated by individual investigator-initiated research funding mechanisms (74%). However, other funding mechanisms were also used, including research program/center grants, individual career development awards, institutional training awards, fellowships, and development awards (Table 2). Respondents were fairly evenly divided among those who reported that they do basic (61%) versus applied (62%) research, with roughly one-quarter (23%) indicating they do both.
Table 2.
Funding Mechanisms and Research Fields
| Percentage of Respondents* | |
|---|---|
| Type of Funding | |
| Research | 74.0 |
| Program Grants | 33.1 |
| Career Development Individual | 20.9 |
| Institutional Training Grants | 12.2 |
| Fellowships | 8.8 |
| Technology Development Grant | 5.7 |
| Other | 10.9 |
| Research Type (all that apply) | |
| Basic | 61 |
| Applied | 62 |
| Both Basic and Applied | 23 |
Does not add to 100% because respondents reported multiple funding types.
The leading basic research fields (not shown) were immunology (56%), cellular biology (46%), molecular biology (38%), physiology (36%), biochemistry (30%), and medicine (30%). The dominant applied research fields were clinical research (60%), translational research (49%), intervention research (32%), and public health research (30%).
Outputs
The survey explored the use of various research dissemination methods (Table 3). Each of the following forms of dissemination was listed by more than 25% of respondents: peer reviewed journal articles, presenting at scientific conferences, participating in grantee meetings, participating in workshops or trainings in which research was disseminated, providing information for press releases, developing and disseminating research tools and methods, presenting research in community forums, and developing and disseminating curricula. Respondents who ever had NIEHS funding, who have a clinical background, or who reported that they do applied science were more likely to use nearly all of these dissemination methods.
Table 3.
Research Dissemination Mechanisms by NIEHS Funding, Clinical Background, and Research Type
| Respondent Characteristics (% respondents) | ||||||||
|---|---|---|---|---|---|---|---|---|
| All | NIEHS | Clinical | Research Type | |||||
| Dissemination mechanism | N=725 | Ever N=192 | Never N=422 | Yes N=353 | No N=334 | Basic N=266 | Applied N=276 | Both N=162 |
| Published in peer-reviewed journals | 94.5 | 95.8 | 93.6 | 96.0 | 94.8 | 96.1 | 92.6 | 95.6 |
| Presented at scientific conferences | 93.5 | 95.3 | 91.7 | 94.8 | 92.9 | 93.1 | 93.4 | 96.3 |
| Participated in grantee meetings | 51.6 | 66.8* | 47.2 | 57.2* | 44.3 | 39.4a | 55.7b | 64.4c |
| Participated in workshops or trainings disseminating your research | 46.9 | 57.4* | 44.7 | 49.7 | 42.8 | 33.2a | 53.5b | 57.5b |
| Provided information for press releases | 30.9 | 45.8* | 27.9 | 34.5* | 24.9 | 15.1a | 39.1b | 42.5c |
| Developed and disseminated research tools and methods | 28.7 | 38.4* | 25.9 | 28.2 | 30.0 | 20.1a | 33.2b | 36.3b |
| Presented research in community forums | 26.7 | 44.7* | 19.8 | 29.6* | 22.2 | 12.7a | 36.9b | 31.3c |
| Developed and disseminated curricula | 26.0 | 36.3* | 23.2 | 29.3* | 21.5 | 18.5a | 32.5b | 27.5c |
| Developed and disseminated interventions | 16.2 | 21.1* | 14.4 | 19.8* | 10.5 | 3.5a | 28.8b | 15.6c |
| Developed fact sheets and pamphlets | 14.9 | 24.2* | 12.7 | 16.7* | 9.5 | 2.3a | 28.8b | 11.9c |
| Developed and published websites | 12.8 | 17.9* | 11.3 | 12.1 | 13.3 | 5.8a | 18.5b | 14.4c |
| Participated in the development of clinical guidelines for the treatment of asthma | 12.0 | 13.2 | 12.7 | 21.3* | 2.5 | 2.7a | 21.4b | 11.3c |
| Provided scientific testimony and briefings to legislators | 9.7 | 19.0* | 6.9 | 12.4* | 5.2 | 5.4a | 13.3b | 11.3b |
The percentage differences are statistically significant (chi-square test, p<.05)
For three-way comparisons, estimates with different lowercase superscripts are significantly different and estimates with the same letter superscript are not significantly different from each other (Tukey–Kramer test, p<.05)
For three-way comparisons, estimates with different lowercase superscripts are significantly different and estimates with the same letter superscript are not significantly different from each other (Tukey–Kramer test, p<.05)
For three-way comparisons, estimates with different lowercase superscripts are significantly different and estimates with the same letter superscript are not significantly different from each other (Tukey–Kramer test, p<.05)
In addition to traditional research dissemination methods described above, investigators have other opportunities to engage with various audiences, either formally or informally (Table 4). Not surprisingly, informal forms of engagement were more commonly reported, although variation by audience is evident. Overall, the results show interaction with the following groups (in descending order): other researchers; university administration/ program directors; local, regional, or national health officials; business and industry representatives; community groups; advocacy groups; environmental regulators; legislators and staffers; food and drug regulators; and housing and urban development agencies. Sharing information ranged from a high of 91% with other researchers to a low of 7% with housing and urban development agencies. A similar pattern was found for conducting joint projects or activities. Serving on boards or advisory panels was most often reported in the context of local, regional, or national health officials (22%); business and industry (20%); or university administration (19%). Formal testimony occurred comparatively infrequently but was most common with legislators and staffers (9%) or with environmental regulators (5%). Employee and consulting arrangements were most common with other researchers (23%) or in business and industry settings (22%).
Table 4.
Nature of Engagement by Audience
| Engagement Type (% respondents) | ||||||
|---|---|---|---|---|---|---|
| Audience | Share information | Conduct joint projects or activities | Serve on boards or advisory panels | Provide formal testimony | Serve as employee or consultant | No interaction |
| Other researchers | 91.0 | 82.9 | 3.1 | 2.7 | 22.8 | 2.0 |
| University administration / program directors | 61.8 | 26.1 | 19.4 | 0.6 | 14.9 | 26.7 |
| Local, regional or national health officials | 41.2 | 16.4 | 22.0 | 4.2 | 10.6 | 48.3 |
| Business and industry representatives | 40.3 | 10.9 | 19.5 | 2.6 | 22.0 | 46.3 |
| Community all | 36.2 | 14.3 | 10.7 | 2.9 | 2.6 | 59.3 |
| Advocacy all | 31.8 | 9.2 | 11.1 | 3.9 | 4.2 | 63.3 |
| Environmental regulators | 19.8 | 4.8 | 11.4 | 4.5 | 4.6 | 73.0 |
| Legislators and staffers | 22.0 | 1.7 | 3.5 | 9.1 | 0.9 | 73.2 |
| Food and drug regulators | 7.6 | 2.2 | 4.8 | 1.6 | 4.0 | 86.8 |
| Housing and urban development agencies | 7.1 | 2.0 | 1.5 | 1.5 | 1.1 | 90.9 |
Those who ever had NIEHS funding, clinical investigators, and applied scientists were more likely than their counterparts to share information with university administration and program directors and with local, regional, or national health officials (not shown). Clinical investigators and those with an applied science background were more likely to share information with food and drug regulators, legislators and staffers, advocacy and community groups. Investigators who ever had NIEHS funding were more likely to share information with environmental regulators, legislators and staffers, business and industry representatives, housing and urban development agencies, and advocacy and community groups.
Among other outputs included in the survey, 17% indicated that their research had led to changes to the curriculum in advanced education settings, and 22% indicated they had applied for a patent. Of the patent applicants, 57% received a patent, and 73% of these reported that federal funding had supported the research that led to the patent.
Outcomes
The survey explored a number of short-term, intermediate, and long-term outcomes (Table 5). The greatest impact was reported in the areas of accumulation of knowledge (intermediate outcome), and replication and new research (short-term outcome). Public advocacy and clinical practice changes were also reported by more than 25% of the respondents. Impacts related to the environment were reported less often, but were significantly more likely to be reported (nearly four-fold) among those who ever had NIEHS funding (not shown).
Table 5.
Survey Self-reported Impacts Related to Model Outcomes: Short-term, Intermediate, and Long-Term
| Outcome | Self-reported Impact Measure (% of respondents) | |
|---|---|---|
| Short-term | Monitoring and surveillance | ■ Improved environmental measurement techniques (20%) |
| Replication and new research | ■ Increased evidence regarding effective interventions (41%) | |
| Commercial products and drugs | ■Started a spinoff or new company (4%) ■Commercialized a patent (19% of patent holders) ■Licensed a patent (38% of patent holders) |
|
| Intermediate | Laws, regulations, and standards | ■Changes in environmental standards or regulations for indoor air (8%) ■Changes in environmental standards or regulations for outdoor air (11%) |
| Guidelines and recommendations | ■ Changes in clinical guidelines for asthma (19%) | |
| Accumulation of knowledge | ■Greater understanding of asthma disease mechanisms (61%) ■Greater understanding of individual, social, and environmental factors and asthma (53%) |
|
| Operations change to reduce environmental hazards | ■Changes in business practices regarding indoor air (8%) ■Changes in business practices regarding outdoor air (9%) |
|
| Public knowledge and attitudes | ■ Changes in public knowledge and practices related to asthma prevention and control (33%) | |
| Long-term | Clinical practice changes | ■ Changes in clinical practice relevant to asthma (27%) |
| Public advocacy | ■ Increased public advocacy for asthma prevention and control (26%) |
End user results
The researchers we surveyed were able to fill in significant detail on the left-hand columns of the NIEHS research impact model (inputs, activities, and outputs). However, we also recognized the importance of engaging end users of asthma-related research in discussions about how the research sponsored by NIEHS and others finds its way into practical, real-world applications (outcomes). End user interviews were considered exploratory to assess the feasibility of using a key informant approach to assess research impact.
End users were identified through literature and web searches, recommendations from an expert panel, formative interviews conducted to help inform survey development, and networking with existing contacts in the asthma field. A total of 16 end users were interviewed by telephone in 2008 representing four domains drawn from Figure 1: business and industry (n=3), clinical practice and guidelines (n=4), education and advocacy (n=4), and regulation (food and drug and environmental) (n=5). Agencies or professional groups that fit within the domain were identified first. Then we identified individuals in the agency who were best positioned to speak to the use of asthma research.
Model pathways
The end users we interviewed confirmed the existence of pathways postulated in Figure 1, and also provided examples of unanticipated pathways. In particular, they identified feedback loops from product development to research (business and industry); from guidelines to product development (business and industry); from products to regulation and clinical practice (clinicians); from public attitudes to clinical practice (education and advocacy); from products to community education and outreach (education and advocacy); and from products and laws, regulations, and standards to clinical practice (regulators). Clinicians also identified the importance of communities of practice as a complement to communities of science in supporting the research-translation process.
Research sources
All end user types were aware of the large, important body of asthma research funded by NIH. Appreciation was expressed for the perception that NIH-sponsored research was less apt to be biased than that funded by commercial sponsors. Most respondents were also aware of which NIH institutes fund asthma research in particular, although some viewed NIH as a monolith and were not necessarily conscious of which institute had funded the research they used.
In their responses to questions about research use, end users discussed where they access research findings, the types of research they use, and how they use it. End users access research through a variety of sources in their respective work (Table 6); some are external sources of information, and others involve information generated in-house (or intramurally). Journals and professional associations figure prominently as external sources of information across all user types. Websites for government agencies, professional associations, and pharmaceutical companies among others are also frequently cited as valuable sources of external information.
Table 6.
Information Sources for Asthma Research Accessed by End Users
| Information Sources | End User Type* | |||
|---|---|---|---|---|
| BI | CL | ED | RG | |
| External Sources | ||||
| Professional journals | √ | √ | √ | √ |
| Professional associations | √ | √ | √ | √ |
| Professional meetings and symposia | √ | √ | √ | √ |
| Pubmed (including alerts) | √ | |||
| Websites (e.g., CDC, HUD, pharmaceutical companies) | √ | √ | √ | √ |
| Internal Sources | ||||
| In-house (intramural) research | √ | √ | ||
| Practice-based research | √ | √ | ||
| Commercial clinical databases from insurers | √ | |||
| Formal meetings with university researchers | √ | |||
| Opinion leader discussions | √ | |||
| Focus all with users of therapeutics | √ | |||
| Feedback from health care providers and sales representatives | √ | |||
Business and Industry [BI], Clinical [CL], Education and Advocacy [ED], Regulators [RG]
√ Indicates that at least one end user in this category identified the information source as one they had used.
Where readily available external sources of information are lacking or are not sufficiently targeted to their needs, end users proactively collect/compile and analyze information on their own. Both the pharmaceutical industry and the federal government have intramural research programs geared toward their unique information needs. Beyond their own in-house research programs, pharmaceutical companies proactively network with university researchers and with those who use their products, such as managed care organizations, patients, health care providers, and sales representatives.
As was apparent from the survey results, the most common research dissemination mode is what can be termed “traditional research products,” including peer-reviewed articles in scientific journals and presentations at scientific meetings and conferences. However, while the end users we interviewed did make use of such traditional products, they also utilized other types of research products. All of the end user types interviewed made use of government reports and data systems in their respective work. Most of those specifically mentioned were related to the drug registration and safety regulation processes. Epidemiological and surveillance data sets and reports available from the CDC and EPA surveillance data related to particulate matter were also mentioned.
End users we interviewed suggested that they value and use abbreviated digests of critical information along with clear recommendations for practice, which are likely to be found in practitioner journal articles, clinical asthma guidelines, literature reviews, and meta-analyses. While the findings reported are evidence-based, they are often presented without extensive methodological discussion. For this type of reporting, “clinical impression” of perceived benefit is considered acceptable evidence in the absence of other more reliable forms of evidence, as was noted by respondents in both drug development and clinical guideline development arenas.
The critical role played by practitioner professional associations in research synthesis and dissemination was also made clear by several respondents, particularly in the clinical and education/advocacy categories. Respondents noted that the professional associations to which they belonged bore the responsibility for educating and training their members and keeping them informed of the latest developments in the asthma field. Practitioner-oriented journals, newsletters, continuing education, training and certification programs, and annual conferences that attract national experts and foster professional networking were identified as supporting the associations in accomplishing this mission.
Research use
Purposes for research use identified by end users fell into the following broad categories: professional development; intervention (including regulation) and remediation; new drug development and regulation; and clinical practice. The types of research respondents found particularly relevant to the latter three uses include a wide range of applied topics (Table 7). Applied topics also predominated in the list of research gaps identified by end users, which was the reverse of the gaps identified by researchers, where basic science topics predominated.
Table 7.
Asthma Research Content Perceived as Useful by End Users
| End User Type* | ||||
|---|---|---|---|---|
| BI | CL | ED | RG | |
| Intervention (including environmental regulation) and Remediation | ||||
| Descriptions and analyses of the social and physical environment | √ | √ | ||
| Triggers and causes of exacerbation | √ | √ | ||
| Environmental controls | √ | √ | √ | |
| Environmental exposures | √ | √ | √ | √ |
| Occupational exposures | √ | √ | √ | |
| Baseline burden in typical home; what is typical and what is high in terms of allergens | √ | √ | ||
| Connections between clinic data and home data | √ | √ | ||
| Issues in building science that affect asthma | √ | √ | ||
| Assessment and remediation of the health effects of contaminants | √ | √ | ||
| Intervention effectiveness | √ | |||
| Social and behavioral influences on case management | √ | |||
| Teaching or learning approaches | √ | |||
| Environmental justice | √ | √ | ||
| Research from all phases in the drug development cycle to support regulation and standard setting | √ | |||
| New Drug Development and Regulation | ||||
| Clinical applications of drugs | √ | √ | √ | √ |
| Effects of medication | √ | √ | √ | √ |
| New medications | √ | √ | √ | √ |
| Therapeutic product information (e.g., tolerance of therapeutics by patients, adverse effects) | √ | √ | √ | √ |
| Changing patterns in asthma prevalence | √ | |||
| Pharmacogenetics | √ | √ | √ | √ |
| Clinical Practice | ||||
| Latest advances in research and implications for practice | √ | √ | √ | √ |
| Etiology of severe asthma and implications for asthma management | √ | √ | ||
| Viral etiologies of asthma and implications for asthma management | √ | √ | √ | |
| Gene-environment interactions | √ | √ | √ | |
| Best practices for managing asthma | √ | √ | √ | |
| Issues in treatment | √ | √ | √ | |
| Issues in adherence | √ | √ | √ | |
| Interactions between different care providers | √ | √ | ||
Business and Industry [BI], Clinical [CL], Education and Advocacy [ED], Regulators [RG]
√ Indicates that at least one end user in this category identified the research content as useful.
Outcomes
We asked end users about their perception/knowledge of the impact of asthma research in the outcome areas of the research impact model about which they were most knowledgeable (Table 8). End users were mixed as to their ability to confidently link asthma research to economic impacts; education/advocacy respondents appeared to be in a better position to speak to this issue than other end user types. Their observations tended to cite specific interventions conducted locally in a hospital or managed care system. The impacts cited fell into the following general categories: decreased work or school absences and fewer dollars spent on medications or on other aspects of health care. Relatively few environmental impacts were identified, although this was largely a function of the particular end users we interviewed. The business and industry representatives among our respondents were involved in drug and device development rather than the control of industrial emissions and so could not comment on the “reduced emissions” pathway in the model. Clinical end users involved in guideline development commented that they were unable to identify much in the way of environmental literature through 2005, the endpoint of their literature searches during the guideline development process. Other end users (e.g., those in the education/advocacy and regulator categories) were able to point to several positive environmental impacts in recent years in the areas of reduced environmental tobacco smoke and reduced environmental exposures (ozone, particulate matter, lead). Respondents were much more sanguine about observed impacts related to health and social areas including: delivery of therapeutics, clinical practice, provider-patient relationships, patient outcomes, and community outcomes. In other words, our sample of end users could readily talk about changes in these outcome categories, including reductions in morbidity and mortality from asthma, but found it difficult to make the connection back to a particular federally sponsored research project.
Table 8.
End User Confirmation of Model Outcomes: Short-term, Intermediate, and Long-Term
| Outcome | End User Type* | ||||
|---|---|---|---|---|---|
| BI | CL | ED | RG | ||
| Short-term | Awareness of ongoing research | √ | √ | √ | √ |
| Policy assessments | √ | √ | |||
| Monitoring and surveillance | √ | √ | √ | ||
| Communities of science | √ | √ | √ | √ | |
| Replication and new research | √ | √ | √ | ||
| Commercial products and drugs | √ | √ | √ | √ | |
| Awareness of environmental health impacts and proposed regulations | √ | √ | √ | √ | |
| Public awareness | √ | √ | √ | √ | |
| Intermediate | Identification of scientific needs/new science | √ | √ | √ | √ |
| Laws, regulations, and standards | √ | √ | √ | √ | |
| Guidelines and recommendations | √ | √ | √ | √ | |
| Accumulation of knowledge | √ | √ | √ | √ | |
| Operations change to reduce environmental hazards | √ | ||||
| Knowledge and attitude change | √ | √ | √ | √ | |
| Long-term | New grant programs | √ | |||
| Improved environment | √ | √ | √ | √ | |
| Reduced human exposure to environmental hazards | √ | √ | √ | √ | |
| Clinical practice changes | √ | √ | √ | ||
| Reduced emissions | √ | ||||
| Public behavior change and advocacy | √ | √ | √ | ||
Business and Industry [BI], Clinical [CL], Education and Advocacy [ED], Regulators [RG]
√ Indicates that at least one end user in this category provided information relative to the outcome, tracing one or more pathway
Summary and discussion
Model validation
This study provided support for the research impact pathways in our model and suggested a few modifications. Specifically, it may be worthwhile to consider adding “communities of practice” along with “communities of science”. Professional organizations and consortia are increasingly playing an interpretive role in vetting, synthesizing, and reducing knowledge so that it can be readily absorbed by busy people into their daily practice. And they play a significant role in community education and awareness, links that were perhaps undervalued in the model. Applying the model to other research programs at NIEHS or other agencies that fund large research programs might identify further modifications to the model.
Researcher survey
From the researcher survey, we obtained information on asthma-related activities that complemented what we could learn from secondary data, especially related to funding. Because the survey used the investigator as the unit of analysis in contrast to the grant, the survey provided a profile of the types of research funding used during an investigator's career. For example, the survey found that non-NIH funding is quite common among these researchers, despite the prominence of NIH as the major funding source. We could also compare investigators who had ever had NIEHS funding versus those who had not. This revealed that those who had received NIEHS funding were more likely to have received program grants and institutional training grants.
The survey was useful for looking at outputs, allowing a more nuanced assessment of dissemination activities and product development. The survey found, for example, that the prevalence of other dissemination modes beyond publications is significant. This suggests that follow-up research to learn more about these non-traditional modes of research dissemination may be warranted. The survey was also able to identify the prevalence of alternative forms of interaction between researchers and non-research groups, interactions that have the potential to lead to research use through non-research pathways.
The contribution to understanding outcomes was more limited. We could identify a range of outcomes and compare researchers’ assessments of their contribution across this wide range of outcomes. However, self-reported impacts have limitations. Researchers can identify changes that have occurred and for which they believe they can take some credit, but they are uncomfortable for the most part with causal statements. They can list areas in which they have made a contribution, such as advancing the science of asthma, but unless they were personally involved, they cannot say with confidence that their work contributed to changes in regulations, guidelines, clinical practice, public knowledge, or changes in business practices. Also, this method does not support attribution of these outcomes to specific research activities nor to specific funding sources. Nor did we gain many insights from the survey into the pathways by which research is translated and used to affect these outcomes.
End user interviews
End user interviews contributed to our understanding in markedly different ways. From our small sample of research end users, we obtained information on research pathways; research awareness and use; and perceived economic, environmental, health and social impacts, as well as ultimate impacts in terms of reductions in morbidity and mortality from asthma. However, making the connection back to a particular federally sponsored research project remained elusive.
End users shed considerable light on research use, but these insights did not relate either to specific investigators or to specific grants. Nor did they necessarily relate to specific funding agencies. More often, their associations were at a higher level, such as NIH, the federal government, or private industry. Only occasionally could an end user cite specific research that they knew had been funded by NIEHS. The connections that end users were able to make with specific agencies tended toward the general rather than the specific. Many of them recognized that NIEHS is more focused on environmental control of asthma and other allergies than on therapeutic treatment. The NIEHS “brand” relating to asthma research is associated with air quality, environmental exposures, and environmental controls.
End user interviews also provided insights into the sources and formats of the research results that they use. Whereas researchers in our sample overwhelmingly favored traditional dissemination modes (peer-reviewed articles in scientific journals and presentation at scientific conferences), end users interviewed sought information about research in a wide variety of formats, many of these involving dissemination modes with a strong practitioner focus. This type of “practice-oriented reporting”—recommended by Nutbeam (1996a, 1996b) and Frenk (1992) as a means to bridge the research-to-practice gap—considers the needs of end users of the information in its content, design, and format. Practical tools and pre-vetted, pre-sorted, and pre-digested information drawing clear implications for practice were valued by the end users we interviewed, and where they could not find these they created them. Some of the end users we spoke with were directly engaged in this translational process.
While no one end user could tell the entire story or trace specific impacts back to specific research products or programs, their collective perceptions suggest that research is used and does produce impacts. Federally funded research is considered of particular value (versus commercially funded research) because it is perceived to be objective and useful for hypothesis generation. The end users themselves play an important role as catalysts for and facilitators of the research translation process. End users also feel they have a potential role to play as partners in the research development process, a role that could be further developed in partnership with federal agencies.
Strengths and limitations of the approach
The model and methods that we applied to an assessment of the NIEHS asthma research portfolio consider a broad range of pathways through which research can impact outcomes. Like Kuruvilla et al. (2006), Trochim et al. (2008), and others, our approach looks at a broad set of research, policy, service, and societal outcomes. Our approach was ambitious in its attempt to look at multiple pathways to these outcomes and to explore the contribution of a range of methods to measure and test the model components, using both secondary and primary data sources.
Limitations included the small and exploratory nature of the interviews with end users of asthma research. We only conducted 16 interviews across 4 domains. We did not, for example, interview legislators, citizens, science journalists, or business and industry representatives responsible for the control of industrial emissions. Additional interviews with more and/or different types of end users could further develop or identify additional insights.
Asthma researchers were asked to self-report the types of impacts they believed their work had. The survey format we used (short, self-administered, online) traded breadth of data collection for depth of insight. It did not allow us to challenge researchers to make the links from specific research activities to outputs and outcomes. Complementary research techniques (e.g., in-person or telephone interviews) with a small set of strategically selected researchers might shed additional light on specific pathways. We do note, however, that the results of developmental interviews with researchers suggest they are relatively conservative about assigning impact to their work. As a result, they are more apt to understate the impacts that may have occurred than to overstate them.
Despite our efforts, and those that preceded ours, the ability to directly attribute NIEHS-supported work to many of the outcome measures remains elusive. The primary data collection activities were better at looking at contribution than attribution. That is, researchers and end users were able to identify changes in model outcomes and in health impacts, often citing specific data sources to back up their claims, but they were on the whole unable to link these changes to specific research activities. They were better at looking at the collective impact of research supported by all federal agencies than they were at teasing out the contributions of a single agency or a single research study.
However, taken together, the results of both the researcher survey and end user interviews indicate that broader impacts from research are occurring and that the model guiding this assessment, with its “pathways,” is a reasonable representation of how research may result in such impacts, at least as they pertain to asthma research. Further research can build on this base by testing the model with other large research programs and by expanding the range of methods used to assess the impacts.
The present study, in combination with the two previous reports (Engel Cox et al., 2008 and Liebow et al., 2009), provides a unique example of working through a science program area logic model both from left to right (inputs to outcomes) and from right to left (from outcomes back to inputs). We have explored the challenges of measuring attribution and contribution of federally funded health research to long-term health outcomes using both primary and secondary data sources. The approaches we used provide advances in several key areas, but there is much work to be done to develop more robust theories, methods, metrics, and data collection systems that will demonstrate clear linkages among research inputs, outputs, and long-term health outcomes.
ACKNOWLEDGMENTS
This article may be the work product of an employee or group of employees of the National Institute of Environmental Health Sciences (NIEHS), National Institutes of Health (NIH); however, the statements, opinions or conclusions contained herein do not necessarily represent the statements opinions or conclusions of NIEHS, NIH or the United States Government. This work was supported under NIEHS Contract Number: HHSP23320045006XI/T.O HHSP233000015T. The authors would like to thank all of the individuals who participated in this study by consenting to an interview or completing a survey. We would also like to thank the following individuals for their material support to the study: Ms. Jane Lambert of NIEHS, and Battelle staff Howard Fishbein, Ed Liebow, Lisa John, Glenna Redmond Wolf, and Morgan Sill. Finally, a note of gratitude to the many internal reviewers and to the anonymous external reviewers whose comments have helped us improve the content and clarity of the manuscript.
References
- Buxton Martin, Hanney Steve. How can payback from health services research be assessed? Journal of Health Services Research and Policy. 1996;1:35–43. [PubMed] [Google Scholar]
- Corley Elizabeth. A use-and-transformation model for evaluating public R&D: Illustrations from polycystic ovarian syndrome (PCOS) research. Evaluation and Program Planning. 2007;30:21–35. doi: 10.1016/j.evalprogplan.2006.09.001. [DOI] [PubMed] [Google Scholar]
- Coryn Chris, Hattie John, Scriven Michael, Hartmann David. Models and mechanisms for evaluating government-funded research: An international comparison. American Journal of Evaluation. 2007;28:437–457. [Google Scholar]
- Cozzens Susan. The knowledge pool: Measurement challenges in evaluating fundamental research programs. Evaluation and Program Planning. 1997;1:77–89. [Google Scholar]
- Engel-Cox Jill, Van Houten Bennett, Phelps Jerry, Rose Shyanika. Conceptual model of comprehensive research metrics for improved human health and environment. Environmental Health Perspectives. 2008;116(5):583–592. doi: 10.1289/ehp.10925. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Frenk Julio. Balancing relevance and excellence: Organizational responses to link research with decision making. Social Science and Medicine. 1992 December;35(11):1397–1404. doi: 10.1016/0277-9536(92)90043-p. [DOI] [PubMed] [Google Scholar]
- Hanney Steve, Frame Iain, Grant Jonathan, Green Philip, Buxton Martin. From Bench to Bedside: Tracing the Payback Forwards from Basic or Early Clinical Research: A Preliminary Exercise and Proposal for a Future Study. HERG Research Report No. 31. Health Economic Research Group, Brunel University; Uxbridge, Middlesex: 2003. [Google Scholar]
- Kuruvilla Shyama, Mays Nicolas, Pleasant Andrew, Walt Gill. Describing the impact of health research: A Research Impact Framework. BMC Health Services Research. 2006;6:134. doi: 10.1186/1472-6963-6-134. [DOI] [PMC free article] [PubMed] [Google Scholar]
- Liebow Edward, Phelps Jerry, Van Houten Bennett, Rose Shyanika, Orians Carlyn, Cohen Jennifer, Monroe Philip, Drew Christie. Toward assessment of the scientific and public health impacts of the National Institute of Environmental Health Sciences extramural asthma research program using available data. Environmental Health Perspectives. 2009;117(7):1147–1154. doi: 10.1289/ehp.0800476. [DOI] [PMC free article] [PubMed] [Google Scholar]
- National Institute of Environmental Health Sciences . NIEHS 2006–2011 Strategic Plan. National Institutes of Health, National Institute of Environmental Health Sciences; Research Triangle Park, NC: 2006. http://www.niehs.nih.gov/external/plan2006. [Google Scholar]
- Nutbeam Don. Improving the fit between research and practice in health promotion: Overcoming structural barriers. Canadian Journal of Public Health. 1996a November;87(Suppl 2):S18–S23. [PubMed] [Google Scholar]
- Nutbeam Don. Achieving “best practice” in health promotion: Improving the fit between research and practice. Health Education Research. 1996b September;11(3):317–326. doi: 10.1093/her/11.3.317. [DOI] [PubMed] [Google Scholar]
- Powers Anne, Rose Shyanika, Bernichon Tiffiny, Engel-Cox Jill, Sill Morgan, Versendaal Kendra. Program Assessments and Evaluation for NIEHS, Work Assignment #1: Literature Review on Research Impact Assessment Methods and Concepts. Battelle Centers for Public Health Research and Evaluation; Arlington, VA: 2006. Final report submitted to the National Institutes of Health, the National Institute of Environmental Health Sciences.
- Quinlan Kathleen, Kane Mary, Trochim William. Evaluation of large research initiatives: Outcomes, challenges, and methodological considerations. In: Coryn CLS, Scriven M, editors. Reforming the Evaluation of Research: New Directions for Evaluation. Vol. 118. 2008. pp. 61–72. [Google Scholar]
- Rubenstein Albert, Geisler Eliezer. The use of indicators and measures of the R & D process in evaluating science and technology programmes. In: Roessner J, Roessner J, editors. Government Innovation Policy. St. Martin's Press; New York, NY: 1988. [Google Scholar]
- Van Houten Bennett, Phelps Jerry, Barnes Martha, Suk William. Evaluating scientific impact. Environmental Health Perspectives. 2000;108:A392–393. doi: 10.1289/ehp.108-a392. [DOI] [PMC free article] [PubMed] [Google Scholar]