Impact of NHLBI-Supported Cardiovascular Epidemiology Research, 1998-2012

Introduction

In a recent paper, Alberts et al. warned that the US-based biomedical science enterprise is flawed in its assumption that the enterprise will constantly expand, and it cannot expect a persistently expanding National Institutes of Health (NIH) biomedical research budget in the future¹. In fact, as noted by multiple observers, the NIH budget has been declining in constant dollars since 2003, and the pay lines for grants are at historic lows^1-4. Such realizations have led to calls for a re-examination of the policies and programs of the NIH and its individual Institutes and Centers^1,2. Some have questioned the appropriateness of the allocation of research budgets between discovery science and translational science^1,2,5. Others have argued for better methods to make funding decisions, suggesting that peer review is too conservative and limits innovation^1,2,4,6,7, or simply does not demonstrate the ability to prioritize research proposals on their potential to yield high impact^8,9. Evidence that scientists spend too much time writing and rewriting grant applications, stay too long in training programs, and achieve their first tenure track position or first NIH grant in their late thirties and early forties, respectively, suggest there is a mismatch in the supply and demand for scientists that must be addressed^1,5,7.

Efforts at the NIH to conduct evaluation and self-examination have already begun. Institutes and Centers within the NIH have taken multiple approaches. The National Cancer Institute (NCI) conducted a workshop to make recommendations on how the NCI can transform itself for the 21^st century to address criticisms of excess expense, repudiated findings, small incremental gains in knowledge, inability to innovate at reasonable cost, and failure to identify research questions with the greatest merit¹⁰. Staff at the National, Heart, Lung, and Blood Institute (NHLBI) have proposed the use of results-based accountability for strategic decision-making to improve the performance of the Institute and the scientists and projects it supports ^11-13. Published evaluations and perspectives to date have focused on funding mechanisms⁸, research designs^{12, 14}, research domains^15,16, clinical trials¹⁷ and research portfolio diversity¹⁸.

In the midst of this evaluation effort to move the NHLBI research portfolio toward “better, faster, cheaper, and more responsive to current needs”¹³, particular attention has been paid to high-cost clinical studies, i.e. clinical trials and epidemiological research^1,2,13. Economic returns of large research studies are being evaluated to determine their value¹⁹. Clinical trials have come under scrutiny for failing to publish results in a timely fashion or, in many cases, not at all¹⁷. Barriers have been identified that keep results from achieving the impact expected by the public^15,17. Epidemiology studies have been criticized for being expensive and outliving their usefulness^1,2; epidemiology has also had to respond to critiques regarding redundancy, false positives, relevance, efficiency, and incremental value^7,12,13,20.

This report offers an impact evaluation of the cardiovascular epidemiology research supported by the NHLBI over a 15 year period (1998-2012). It attempts to address three questions: 1) Over this period, what has been the contribution of NHLBI support to high-impact cardiovascular epidemiology research? 2) What are the trends in the contribution of NHLBI support to high-impact cardiovascular epidemiology research over multiple domains to which epidemiology is expected to contribute, and are there differences in these trends by research domain? 3) Does the contribution of NHLBI support extend to the papers appearing in the most-represented journals publishing high impact cardiovascular epidemiology research?

Methods

Unlike many of the research areas of the NIH, epidemiology is methods-based rather than disease- or organ systems-based. As such, the discipline often crosses multiple diseases and many health topics. Ascertainment of what is a product of epidemiology research is challenging because reports from epidemiology studies do not always use the word “epidemiology” in titles, abstracts or keywords associated with the publications. As a result, a search of epidemiology will not always include all the articles resulting from this discipline.

To address this challenge, the authors developed a list of ten research domains to which cardiovascular epidemiology is likely to contribute. For each domain, a search of the biomedical literature between 1998 and 2012 was performed using SCOPUS (http://www.elsevier.com/online-tools/scopus) during the week of January 24-30, 2014. The search terms used for each of the ten research domains selected to represent cardiovascular epidemiology research are presented in Table 1. The search within each domain included the search terms specific to that domain plus the additional terms presented at the bottom of the table that were common to all domains. This approach provided a degree of consistency across domains. The last term “AND NOT randomiz*” was added in order to eliminate randomized clinical trials from the results. The domain “New Biology” represents a particular point in scientific progress. As these individual fields within “New Biology” (defined by search terms in Table 1) become integrated into current knowledge, this domain will either have to be reassessed for validity or noted as being temporally defined when the impact of future funding is determined. Average citations per year were calculated based on the year of publication and the total citations identified through the end of 2013. The results of each search were sorted by average number of citations per year.

Table 1. Domain-Specific Search Terms Used for SCOPUS Publications Search.

Behavioral Research	“adolescent behavior” OR “child behavior” OR “health behavior” OR “risk reduction behavior” OR “social behavior” OR “social psychology” OR “social adjustment”
Co-morbidity/Syndromes	comorbid OR syndrome OR co-morbid
Genetics	genetic* OR genomic* OR gene
Health Disparities	“health status disparities” OR “healthcare disparities” OR “social class” OR “socioeconomic status” OR “socioeconomic factors” OR “factors,socioeconomic” OR “minority health”
Methodology	epidemiology AND methodology
New Biology	proteomic* OR metabolomic* OR microbiome* OR epigenetic* OR RNA OR imaging OR exosome* OR “systems biology”
Pre-clinical Disease	“preclinical OR pre-clinical OR subclinical OR sub-clinical”
Quality of Care/Clinical Epidemiology	“quality of care” OR “clinical epidemiology”
Risk Factors	“risk factor*”
Surveillance	surveillance OR prevalence OR incidence
Additional Search Terms for all Domains	AND (cardiovascular OR coronary) AND (human* OR “public health” OR clinical OR population* OR cohort* OR epidemiol*) AND NOT randomiz*

In each domain, the top 100 publications based on average citations per year were selected and evaluated for authorship, funding support, and acknowledgments to determine if the publications involved any support by the NHLBI.

NHLBI support for a publication was based on satisfying at least one of the criteria presented in Table 2. Determination of NHLBI support was most easily identified for those publications citing funding source with the grant number beginning with “HL”. Publications acknowledging “NHLBI” or “National Heart, Lung, and Blood Institute” were also straightforward. Those supported by contract funding were more difficult to identify because of a shift in contract number coding system during this time period (shift from a 7-character code to a 17-character code in ∼2004 http://www.hhs.gov/regulations/hhsar/subpart304-70.html). Publications derived from projects that were primary or ancillary to a cohort study funded by the NHBLI were recognizable when they cited the parent study by name (e.g. MESA, CARDIA) but some did not. Those publications using data from an NHLBI study obtained through the NHLBI data repository BioLINCC (https://biolincc.nhlbi.nih.gov/home/) or dbGaP (http://www.ncbi.nlm.nih.gov/gap) were more challenging to identify if they did not acknowledge the data repository or, in the case of dbGaP, the specific study. Papers using NHANES data not otherwise coded as NHLBI-supported (see Table 2) were included if they used blood pressure or lipid data, as those components were specifically supported by NHLBI during this time interval. Publications were also considered NHLBI-supported if they acknowledged relying on tools developed by NHLBI-supported studies or assistance by NHLBI staff, but these were few. Finally, publications that were silent on funding source were coded as unknown. After coding, the percent of publications supported by NHLBI was calculated for each research domain. Results were summarized graphically to address question 1. In addition, because the research domains are not mutually exclusive, the top 100 publications across all domains were pooled and duplicate publications were eliminated; the percent with NHLBI support was calculated for the pooled but not duplicated publications.

Table 2. Criteria Sufficient to Code a Publication as NHLBI-Supported.

One or more of the following: Cited grant/contract funding from NHLBI Co-authorship by NHLBI employee Use of NHLBI-funded cohort data or samples Use of NHLBI-supported BioLINCC or dbGaP shared data sets Proceedings of NHLBI sponsored workshops Acknowledged use of tools produced by NHLBI Acknowledged assistance of NHLBI staff Use of NHANES data from NHLBI-funded components (if not otherwise coded as NHLBI-supported): Blood pressure Lipids

To address question 2, the top 100 publications for each domain were broken into three successive 5-year intervals: 1998-2002, 2003-2007, and 2008-2012. The percent of NHLBI-supported publications within each time interval was calculated based on the above methods, and the trends were graphed for each domain. The proportions of papers supported by NHLBI across the three time intervals for each domain were tested to determine if they differed significantly using a chi square test with two degrees of freedom. For those that were significant, a second test for linear trend was conducted with a chi square test with one degree of freedom. An additional analysis following a comparable process was conducted for the pooled unique publications across all domains.

For question 3, the top 100 publications for each domain were sorted by journal. The top 5 journals by frequency of top 100 publications for each domain were identified and results summarized.

Results

The total numbers of publications identified in the ten domains over the 15-year period are summarized in Figure 1, using a logarithmic scale since the total number of publications across the selected domains varied by as much as two orders of magnitude. Fewest papers were published for the domain “methodology” with only 643 papers published over the 15-year period. The most papers were published for the domain “risk factors” with 86,017 papers published over the same time period. In total, more than 225,000 “epidemiology papers” were published in these ten domains during the period 1998-2012.

Figure 1.

Total Number of Publications Identified From Domain-Specific Searches.

Over the same period, 82.5% of the top 100 publications over the ten domains provided sufficient information to code NHLBI support. Figure 2 presents the percent of the top 100 publications within each research domain with NHLBI support based on average citations per year. Percent of publications with NHLBI support varied from a low of 12% for the domain “quality of care/clinical epidemiology” to a high of 46% for “risk factors”. The median proportion of publications with NHLBI support over the ten research domains was 25%. The mean percent of NHLBI-supported publications was 26.8% across the ten domains.

Figure 2.

Percent with NHLBI Support of Top 100 Most-Cited Publications by Research Domain (Published 1998-2012; Citations through 2013).

When all the publications were pooled and duplicates were eliminated, the 1000 publications for ten domains were reduced to 845. The percent of NHLBI-supported publications among non-duplicative publications was 25.6%.

Figure 3 presents trends in the percent of publications with NHLBI support over the three successive 5-year intervals for each of the ten research domains. Trends suggested overall increases in five domains and overall decreases in five, but percentages were statistically significantly different among time periods for only two domains: genetics (p=.039) and quality of care/clinical epidemiology (p=.024). When tested for linear trend, only quality of care/clinical epidemiology showed a statistically significant increasing trend (p=.028). Again, pooling the data and eliminating duplicates provided an overall measure of the trend for NHLBI-support of high impact cardiovascular epidemiology research papers and indicated no significant change over the three time periods (p=.056).

Figure 3.

Trends in Percent with NHLBI Support of Top 100 Publications by Average Citations/Year within Research Domain by 5-Year Periods (1998-2012).

Finally, Table 3 presents the five most-represented journals within the top 100 publications for each domain. Additionally, Table 3 shows how many of the top 100 papers are in those five journals, and how many of those are supported by NHLBI. Among all domains, the most frequently appearing journals (in order) are Circulation, JAMA, NEJM, JACC and JAMA Internal Medicine. Within each domain, the five most-represented journals published a third or more of the top 100 publications in most domains and 50% or more of the top 100 publications in four domains: risk factors, genetics, comorbidity/syndromes, and new biology. High-impact papers were most concentrated in fewer journals for genetics and risk factors where only 28 and 23 journals, respectively, accounted for the top 100 published papers; methodology was the most diffuse where 71 journals accounted for the top 100 published papers. The proportion of NHLBI-supported papers in the five most-represented journals of each domain compared to the proportion in the top 100 papers of that domain suggests the NHLBI-supported papers are equally or more represented among the papers in the most-represented journals.

Table 3. Five Most Represented Journals in Top 100 Publications by Average Citations/Year By Research Domain (1998-2012).

Domain	Total # Journals for 100 papers	Five Most Represented Journals					% Publications with NHLBI Support
		First	Second	Third	Fourth	Fifth	Among the Five Most Represented Journals@	Among 100 Most Cited Publications
Behavioral Research	56	Circulation	JAMA	The Lancet	JAMA Intern Med	JACC; MMWR Surv	15/36;15/4141.7%;36.6%	28%
Co-morbidity/Syndromes	36	Circulation	NEJM	JAMA	JAMA Intern Med	European HJ; JACC*	20/54;21/58*37%;36.2%	28%
Genetics	28	Nature	Circulation	Nature Genetics	Science	NEJM	20/6232.3%	29%
Health Disparities	44	JAMA	The Lancet	JAMA Intern Med	Am J Epidemiol	Br Med J; Circulation; NEJM*	9/42;9/48;10/54*21.4%;18.8%;18.5%	24%
Methodology	71	PLoS One	Am J Epidemiol	Stroke	Circulation	Diabetologia; Int J Epidemiol*	10/23;11/26*43.5%;42.3%	17%
New Biology	35	Circulation	JACC	Nature	European Heart J	NEJM; Science*	10/51;12/56*19.6%;21.4%	24%
Pre-clinical Disease	49	Circulation	JACC	JAMA	JAMA Intern Med	Annals Int Med	25/4654.3%	42%
Quality of Care/Clinical Epidemiology	43	Circulation	JACC	JAMA	Am Heart Journal	Am J Med; NEJM*	8/39;8/44*20.5%;18.2%	12%
Risk Factors	23	NEJM	Circulation	JAMA	The Lancet	JACC	34/7346.6%	46%
Surveillance	55	The Lancet	JAMA Intern Med	Circulation	JAMA	NEJM; MMWR Surv	9/34; 9/3926.5%;23.1%	21%

^@Ratios represent the number of NHLBI-supported papers/number of papers in the top five most-represented journals. The percent is the ratio re-expressed.

^*Indicates a tie for fifth most frequent journal in the top 100 publications.

For this analysis, 133 papers were reviewed by two or more coders to determine NHLBI support because they appeared in multiple domains with different coders. This masked comparison enabled an estimate of coding errors. Analyses of these duplicates, counting all possible pairs when more than two coders provided ratings, indicated 95.6% agreement among coders on NHLBI support.

Discussion

This analysis indicates that, based on citation analysis, at least one quarter of the high-impact cardiovascular epidemiology research papers of the last 15 years were supported, at least in part, by NHLBI. While an expectation for that percentage is difficult to determine, the estimate of about 25% overall is higher than might be expected given that the proportion of biomedical research and development expenditures by the public sector in the United States in 2012 was $48.9 billion or 18.2% of the $268.4 billion (in US dollars) spent worldwide³, of which NIH represents about $30 billion or 11.2% and NHLBI represents a little more than $3 billion or 1.1%.

The 15-year time period was chosen to ensure sufficient time to obtain stable estimates for statistics and to evaluate recent trends within domains. Trend analyses did not indicate substantial change in the impact of NHLBI support for most research domains over the 15-year period. Only genetics and quality of care/clinical epidemiology showed significant variation over the three intervals with only the latter showing a significant upward linear trend in the impact of NHLBI support.

There are several implications of these analyses. First, NHLBI-supported cardiovascular epidemiology research accounts for about one-quarter of the top 100 high-impact papers published over the last 15 years. Considering that cardiovascular epidemiology is supported by many other government agencies and foundations, both in the US and abroad, the NHLBI support of about one-quarter of the highest cited epidemiology research is significant and suggests it is using its resources wisely. Second, the impact of NHLBI-supported research in “quality of care/clinical epidemiology” started from a low base but has significantly increased over the 15-year period and may reflect growing emphasis within the NHLBI on translation²¹. Nevertheless, this domain still ranks lowest for percent with NHLBI support among the ten domains studied. Detailed review of the “quality of care/clinical epidemiology” publication support data indicates this domain intersects the research missions of other Institutes within the National Institutes of Health; other federal agencies including the Agency for Healthcare Research and Quality and the Veteran's Administration; and European countries, the U.K., and Canada as well as the research agenda of the pharmaceutical industry. Third, although the linear trend was not statistically significant, the doubling of the percentage of high-impact genetics papers supported by NHLBI in the last 5-year period from about 20% to 40% also likely reflects the NHLBI's major investment in genetic research and the opportunity researchers have had to take advantage of its longstanding cohorts with stored samples and a wide array of phenotypes²². Finally, the levels and stability of the remaining research domains may be useful to the NHLBI in its decision-making about research directions and priorities going forward.

This analysis has a number of strengths. It covers a 15-year period that is of sufficient length to obtain stable estimates of the contribution of NHLBI support to high impact cardiovascular epidemiology research. It includes ten domains that offer multiple perspectives on the contribution of NHLBI-supported cardiovascular epidemiology research. It was conducted with few restrictions on the type of research support provided.

The analysis also has a number of limitations. An assessment of the impact of NHLBI-supported epidemiology research is not straightforward and, by necessity, must be captured indirectly. Attempts were made to capture most, if not all, the highest-impact publications in cardiovascular epidemiology over the period of study and to identify those with NHLBI support. Impact was measured by average annual citations that favor more recent publications over earlier publications in the time interval. Alternative metrics were considered, such as the average number of citations during the first five years, but currency of the data was considered a higher priority than consistency of the follow-up period. The list of journals where high impact papers were published favored those that have existed over the entire time interval examined and this may change with time as many new journals have been introduced during the 1998-2012 time period. Unknown support information for 16.9% of publications may have altered the estimates of NHLBI support, although the estimates were conservatively calculated as if all support information were known; therefore, the true impact of NHLBI-supported cardiovascular epidemiology research may be even greater. Finally, self-citations were not excluded from this examination. To investigate this effect, two domains were evaluated for the impact of self-citation, again using SCOPUS, and at most 8% of the papers changed, all within the bottom quintile of the top 100 papers. NHLBI support changed at most by 1%.

In summary, this analysis represents an attempt at an unbiased evaluation of the impact of cardiovascular epidemiology research funded by the NHLBI over the period 1998-2012. When data were evaluated multiple ways and over multiple domains, the results were fairly consistent that about 25% of the highest impact cardiovascular epidemiology research publications, as measured by citation analysis, involved support, at least in part, by the NHLBI. While expectations are difficult to calculate, the proportion of high-impact epidemiology research supported by the NHLBI is greater than budgetary data would suggest. These findings will inform future directions for NHLBI-supported epidemiology research and provide a baseline for future analyses to judge the comparative effects of any change in funding priorities at the NHLBI.