Abstract
Introduction:
This paper provides the first detailed analysis of the NIH prevention research portfolio for primary and secondary prevention research in humans and related methods research.
Methods:
The Office of Disease Prevention developed a taxonomy of 128 topics and applied it to 11,082 projects representing 91.7% of all new projects and 84.1% of all dollars used for new projects awarded using grant and cooperative agreement activity codes that supported research in fiscal years 2012–2017. Projects were coded in 2016–2018 and analyzed in 2018.
Results:
Only 16.7% of projects and 22.6% of dollars were used for primary and secondary prevention research in humans or related methods research. Most of the leading risk factors for death and disability in the U.S. were selected as an outcome in <5% of the projects. Many more projects included an observational study, or an analysis of existing data, than a randomized intervention. These patterns were consistent over time.
Conclusions:
The appropriate level of support for primary and secondary prevention research in humans from NIH will differ by field and stage of research. The estimates reported here may be overestimates, as credit was given for a project even if only a portion of that project addressed prevention research. Given that 74% of the variability in county-level life expectancy across the U.S. is explained by established risk factors, it seems appropriate to devote additional resources to developing and testing interventions to address those risk factors.
INTRODUCTION
Established in 1986, the Office of Disease Prevention (ODP) is responsible for assessing, facilitating, and stimulating research in disease prevention and health promotion across the NIH (https://prevention.nih.gov). ODP collaborates with all 27 Institutes and Centers (ICs) at NIH to achieve its mission: to improve public health by increasing the scope, quality, dissemination, and impact of prevention research supported by NIH. ODP requires a detailed understanding of the portfolio of funded prevention research projects to track levels and trends in funding in prevention research over time and to identify research areas that may benefit from targeted efforts by NIH.
ODP defines prevention research to include primarya and secondaryb prevention research in humans and studies of prevention-related methods for use in humans. The ODP definition excludes basic and preclinical studies as too distal to be considered primary or secondary prevention; Calitz et al.1 made a similar distinction. The ODP definition also excludes treatment studies, with five exceptions. Treatment for smoking, drug abuse, alcohol abuse, weight loss treatment for overweight/obese populations, and preventing those with suicidal ideation or behaviors from committing suicide are considered secondary prevention.
ODP proposed to develop enhanced portfolio analysis tools and processes in its 2014–2018 strategic plan.2 This paper describes those tools and processes and presents the results of applying those tools and processes to characterize new research projects awarded as grants and cooperative agreements for fiscal years 2012–2017 (FY12–17).
METHODS
Taxonomy Development
ODP developed a prevention research taxonomy to guide coding of research projects. The taxonomy consists of six categories: rationales, exposures, outcomes, populations, designs, and types of prevention research. Within each category are topics (Figure 1); one or more topics from each category may be selected for a project.
Fig. 1.
Abstract coding form.
The rationale category reflects the health condition(s) (diseases, disorders, injuries, or disabilities) that motivated the study, with 23 topics including many of the leading causes (www.cdc.gov/nchs/fastats/leading-causes-of-death.htm) and risk factors3 for death and disability in the U.S. The exposure category reflects the independent variable(s) in the study, with 40 topics including most of the conditions used for the rationale and many other exposures of interest in prevention research. The outcome category constitutes the dependent variable(s) in the study, with 41 topics that overlap almost completely with the exposure topics. The population category represents the population groups being studied, with 11 topics. The design category indicates the methodology employed and has seven topics and the prevention research category reflects the type of prevention research, with six topics. Project staff were trained to select all topics that applied for a given project and it was common to have multiple selections in many of the categories.
Identification of Activity Codes
The portfolio analysis was restricted to the NIH extramural (conducted by researchers at institutions outside of NIH) research portfolio because it accounts for >80% of all NIH spending; in addition, intramural (conducted by researchers who are employed by NIH) research abstracts are structured as progress reports whereas extramural research abstracts are structured as proposals, making it difficult to use the same methods for both sets of abstracts. To determine which extramural projects to include, ODP consulted with staff from many of the NIH ICs to identify activity codes likely to support NIH prevention research. NIH uses >200 activity codes (https://grants.nih.gov/grants/funding/ac_search_results.htm) to identify extramural funding programs, but many are used for purposes other than research. Activity codes used for community services, facilities, infrastructure, loan repayment, meetings, planning, and training were excluded. Projects supported by contracts were excluded because contract abstracts often provided little insight into the nature of the work.
Extramural research is concentrated in the R (research grants), P (program project grants), and U (cooperative agreements) activity codes. Twelvec activity codes from these categories were included in the portfolio analysis and each had >500 awards during FY12–17 or >$500 million in awards during FY12–17. To avoid double counting, this study included only new awards (type 1), renewals (type 2), and new awards that changed institutes or centers before the project had begun (type 9) that were made using these 12 activity codes. For activity codes that included both a parent project and subprojects (e.g., large program project grants), sampling was done at the subproject level only, also to avoid double counting. For this reason, the units of coding and analysis are referred to as projects.
Projects were identified from NIH’s internal grant database, IMPAC II, using the NIH Query/View/Report system (https://archives.nih.gov/asites/era/11–25-2015/nih_and_grantor_agencies/other/query_view_and_report.cfm) to ensure that all type 1, 2, and 9 awards made in FY12–17 using the 12 selected activity codes were included.
Identification of Likely Prevention Projects
ODP used machine learning (ML) tools to predict prevention research and non-prevention research projects; details on the ML tools are provided elsewhere.4 The ML algorithms were trained using the previous FY’s manually coded type 1 R01d projects (training set), and the trained algorithms were used to classify each project in the FY of interest (test set). Because of random variability in the performance of the algorithms, each algorithm was trained and tested five times, each time with a new starting point.
The initial plan was to code only type 1 R01s; as a result, 100% of the type 1 R01s that were classified as prevention research together with a 5% random sample of the remaining type 1 R01s for FY12–17 were coded. ODP later decided to expand the scope of the portfolio analysis to include 11 other activity codes and two other types. To limit the workload, only 50% of those prevention projects and 5% of those non-prevention projects, as identified by the ML algorithms, were coded. All projects were coded in a random order within activity code and FY, and coding staff were blind to the ML classifications.
Coding Research Projects
ODP developed a coding form (Figure 1) based on the prevention research taxonomy and a detailed coding protocol that provides instructions, definitions, and examples (Appendix). Projects were coded using a team-based, consensus-driven approach using the project’s title, abstract, and public health relevance sections, hereafter referred to collectively as the project’s abstract. To ensure accurate coding across a wide range of prevention research topics, all project staff completed an 8-week training program and met standard performance criteria before coding. Staff had to achieve 70% agreement with the answer keys for multiple sets of training abstracts.
Each member of a team of three research analysts (RAs) read the project’s abstract and selected the applicable topics to characterize the project, with instructions to select all topics within each category that applied to each project. The team members then discussed their coding and reconciled differences to generate a set of consensus codes. When a consensus could not be reached, the supervisor was called to adjudicate. If the issue persisted, the project was added to the sample selected for quality control (below). There were enough RAs to maintain two to three teams and the composition of the teams was randomized daily to avoid consistent triads.
Quality Control
Every week, 10%–20%e of the RA-coded projects were selected at random and, together with any projects for which the RAs were unable to achieve consensus, were coded independently by three ODP staff blinded to the RA coding, using the same coding process and software. After the ODP team reached consensus, the ODP team compared their consensus to the RAs’ consensus, and a final set of codes was established for that project, with weekly discussions between the two teams. This identified problem areas and ensured that the teams were continually aligned over time. The final codes reported for a given project were the consensus codes from the RAs, or, if available, the final consensus codes after the ODP quality control review.
Statistical Analysis
Kappa statistics were calculated for each of the categories to evaluate consistency among RAs and between the RAs and the ODP staff. The method accommodated selection of multiple topics within categories5 and was implemented using SAS, version 9. Weighting and extrapolation from the sample to the population of projects was completed in Stata/SE, version 15, using the svyset command with the pweight and finite population correction options. Weights were calculated as the number of projects funded in each (fiscal year X activity code X type) combination divided by the number of projects coded in each combination, computed separately for prevention and non-prevention projects.
Tests for linear trend were applied to each of the 128 topics, using a Bonferroni corrected criteria of p=0.05/128=0.00038. Analysis was completed in 2018.
RESULTS
Interrater Agreement
Coding was completed between October 2015 and February 2018. The monthly weighted average Κs between the two sets of consensus codes was 0.86 (range, 0.67–1.00).
Coverage
During FY12–17, NIH issued 63,381 awards for projects using R, P, and U activity codes that support research, valued at $32.6 billion. The 12 activity codes selected by ODP represented 58,104 (91.7%) of these awards and $25.7 billion (84.1%) of these dollars.
Sensitivity and Specificity for Prevention
Prevention and non-prevention projects were selected for coding and weighted for analysis as described above. The ML ensemble had a sensitivity of 70.3% and specificity of 91.1% for identifying prevention projects using the ODP definition of prevention.
Projects and Dollars by Topic and Fiscal Year
Figure 2 presents estimated proportions of NIH projects and dollars used for prevention research during FY12–17. It is estimated that of all projects and dollars NIH used for research grants and cooperative agreements, 16.7% of projects and 22.6% of dollars were used to support primary and secondary prevention research in humans or for methods studies to support prevention research; 3.2% of projects included prevention-related methods research without other primary or secondary prevention research in humans. In FY2014, there was a sharp increase in dollars used for cooperative agreements that supported prevention research related to clinical trial networks.
Fig. 2.
(A). Estimated proportion of NIH projects for grants and collaborative agreements that were used to support primary and secondary prevention research in humans, as well as prevention related methods research, Fiscal Year 2012–2017. (B). Estimated proportion of NIH dollars for grants and collaborative agreements that were used to support primary and secondary prevention research in humans, as well as prevention related methods research, Fiscal Year 2012–2017.
Table 1 summarizes the results for coding projects based on their rationale; topic definitions are provided in the protocol (Appendix). Mortality was selected as a rationale more often than any other topic. Cancer, infectious diseases, maternal/paternal/child health, heart disease, mental health, stroke, substance use disorder, neurologic disease, and obesity made up the rest of the top ten rationale topics. During FY12–17, cancer declined as a rationale, from 22.1% in FY12 to 11.2% in FY17 (p=0.00004). None of the other rationale topics presented a significant trend over FY12–17.
Table 1.
Estimated Proportion of Projects by Topic and Fiscal Year (FY) Based on Their Rationalesa.
| Rank | Topic | FY2012 % (95% CI) |
FY2013 % (95% CI) |
FY2014 % (95% CI) |
FY2015 % (95% CI) |
FY2016 % (95% CI) |
FY2017 % (95% CI) |
Average % (95% CI) |
p-trendb |
|---|---|---|---|---|---|---|---|---|---|
| 1 | Mortality | 31.3 (26.1, 36.9) |
22.3 (18.3, 26.8) |
34.1 (28.9, 39.8) |
27.7 (23.3, 32.6) |
28.3 (24.4, 32.4) |
23.2 (19.6, 27.2) |
28.0 (26.1, 30.0) |
0.09415 |
| 2 | Cancer | 22.1 (17.5, 27.6) |
20.0 (15.3, 25.7) |
20.7 (16.3, 25.9) |
17.4 (14.3, 20.9) |
15.7 (13.0, 18.7) |
11.2 (8.2, 15.0) |
17.8 (16.2, 19.7) |
0.00004 |
| 3 | Infectious disease | 17.5 (13.7, 22.0) |
14.3 (11.2, 18.1) |
21.7 (17.2, 27.0) |
19.5 (15.7, 23.9) |
16.6 (13.2, 20.7) |
16.7 (13.7, 20.1) |
17.8 (16.2, 19.6) |
0.95843 |
| 4 | Other or unclear | 15.9 (12.0, 20.8) |
16.1 (12.0, 21.3) |
12.1 (9.5, 15.4) |
17.7 (13.5, 22.9) |
14.0 (10.7, 18.0) |
14.9 (10.9, 20.0) |
15.1 (13.5, 16.9) |
0.77814 |
| 5 | Maternal/paternal/ child health |
12.2 (9.4, 15.6) |
13.1 (10.0, 16.9) |
12.9 (10.0, 16.4) |
11.4 (9.2, 14.2) |
13.6 (10.7, 17.1) |
16.2 (12.8, 20.4) |
13.2 (12.0, 14.6) |
0.14479 |
| 6 | Heart disease | 9.3 (6.8, 12.6) |
9.8 (7.0, 13.6) |
9.3 (6.8, 12.6) |
13.0 (9.7, 17.3) |
11.5 (8.7, 15.1) |
8.6 (6.7, 11.0) |
10.2 (9.0, 11.5) |
0.67808 |
| 7 | Mental health | 10.4 (7.8, 13.8) |
14.7 (10.7, 20.0) |
6.0 (4.8, 7.4) |
8.2 (6.7, 9.9) |
11.3 (8.4, 14.9) |
11.4 (8.8, 14.7) |
10.2 (9.0, 11.4) |
0.91208 |
| 8 | Substance abuse | 8.0 (6.6, 9.7) |
8.4 (6.2, 11.2) |
8.3 (6.0, 11.4) |
9.9 (7.7, 12.7) |
6.8 (5.5, 8.4) |
12.7 (9.4, 17.0) |
9.1 (8.1, 10.2) |
0.05303 |
| 9 | Stroke | 7.3 (5.3, 9.9) |
7.5 (5.4, 10.5) |
8.8 (6.1, 12.6) |
10.5 (7.5, 14.4) |
9.0 (6.8, 11.8) |
8.2 (6.3, 10.6) |
8.6 (7.5, 9.8) |
0.31031 |
| 10 | Neurological disease (not Alzheimer’s) |
5.9 (4.1, 8.6) |
5.1 (3.2, 8.1) |
7.9 (5.1, 12.0) |
9.0 (6.2, 12.9) |
9.8 (7.0, 13.4) |
9.8 (7.0, 13.6) |
8.0 (6.8, 9.3) |
0.00480 |
Tabled values are the estimated proportions and 95% CI of NIH projects funded as grants or cooperative agreements that supported primary and secondary prevention research in human, as well as related methods research for FY2012–2017, based on the rationales coded for each project. The topics are ranked based on average proportion of prevention projects coded for each rationale and the table presents the top ten; the complete table is provided in Appendix Table 1.
With 128 tests for trend, the test is significant only if p-trend<0.00039.
Table 2 summarizes the results for coding projects based on their exposures. Genetics was selected as an exposure for 26.9% of prevention projects, reflecting the large number of projects that included genetic-risk studies, including genome-wide association studies. Education/counseling, medication/device, diet/nutrition, healthcare delivery, infectious disease, chemical/toxin, physical activity, tobacco, and substance use disorder made up the remaining top ten exposure topics. Topics like infectious disease, chemical/toxin, tobacco, substance use disorder, and mental health are used as exposures in studies that group participants by their prior history for these topics and examine their risk for onset of a new condition or progression of an existing condition. None of the exposure topics presented a significant trend over FY12–17.
Table 2.
Estimated Proportion of Projects by Topic and Fiscal Year (FY) Based on Their Exposuresa.
| Rank | Topic | FY2012 % (95% CI) |
FY2013 % (95% CI) |
FY2014 % (95% CI) |
FY2015 % (95% CI) |
FY2016 % (95% CI) |
FY2017 % (95% CI) |
Average % (95% CI) |
p-trendb |
|---|---|---|---|---|---|---|---|---|---|
| 1 | Other or unclear | 56.9 (51.6, 62.1) |
52.7 (46.9, 58.4) |
52.3 (46.7, 57.7) |
57.7 (52.5, 62.7) |
59.8 (55.2, 64.2) |
55.1 (49.8, 60.2) |
55.7 (53.5, 57.8) |
0.48771 |
| 2 | Genetics | 27.7 (22.8, 33.2) |
31.0 (25.5, 37.0) |
24.6 (20.1, 29.7) |
30.1 (25.1, 35.6) |
25.9 (21.8, 30.5) |
22.6 (18.8, 27.0) |
26.9 (24.9, 28.9) |
0.09479 |
| 3 | Education/ counseling |
12.9 (10.9, 15.1) |
13.3 (10.6, 16.5) |
15.3 (12.5, 18.6) |
12.9 (11.0, 15.1) |
15.7 (13.5, 18.1) |
15.3 (12.4, 18.7) |
14.3 (13.2, 15.4) |
0.14592 |
| 4 | Medication/device | 6.9 (4.6, 10.2) |
7.0 (4.4, 10.8) |
15.7 (11.5, 21.2) |
8.8 (6.3, 12.2) |
7.3 (5.9, 9.0) |
11.9 (8.2, 17.0) |
9.8 (8.4, 11.4) |
0.19599 |
| 5 | Diet/nutrition | 6.5 (4.5, 9.1) |
3.9 (2.9, 5.1) |
5.7 (3.6, 8.9) |
4.4 (3.4, 5.7) |
4.9 (3.8, 6.3) |
5.6 (3.9, 8.2) |
5.2 (4.5, 6.0) |
0.76013 |
| 6 | Healthcare delivery | 5.8 (4.0, 8.2) |
3.3 (2.4, 4.4) |
4.3 (2.7, 6.6) |
3.7 (2.8, 4.8) |
3.6 (2.7, 4.7) |
4.9 (3.2, 7.4) |
4.3 (3.6, 5.0) |
0.61019 |
| 7 | Infectious disease | 3.3 (1.8, 6.1) |
2.0 (1.4, 2.9) |
5.9 (3.7, 9.4) |
3.6 (2.7, 4.7) |
4.4 (2.6, 7.3) |
3.1 (2.3, 4.2) |
3.8 (3.1, 4.6) |
0.69820 |
| 8 | Policy/built environment |
3.6 (2.7, 4.7) |
3.8 (2.8, 5.0) |
2.7 (2.0, 3.6) |
2.8 (2.1, 3.8) |
4.6 (3.6, 5.9) |
4.7 (3.6, 6.0) |
3.7 (3.3, 4.1) |
0.07446 |
| 9 | Chemical/toxin | 2.6 (1.8, 3.6) |
3.7 (1.9, 6.8) |
4.3 (2.4, 7.5) |
3.9 (2.9, 5.2) |
2.8 (2.1, 3.9) |
2.8 (2.0, 3.8) |
3.4 (2.8, 4.1) |
0.73134 |
| 10 | Tobacco | 2.4 (1.7, 3.4) |
5.2 (2.9, 9.3) |
2.3 (1.0, 5.1) |
1.5 (0.9, 2.3) |
2.3 (1.7, 3.2) |
2.4 (1.7, 3.4) |
2.6 (2.1, 3.4) |
0.08577 |
Tabled values are estimated proportions and 95% CI of NIH projects funded as grants or collaborative agreements that supported primary and secondary prevention research in humans, as well as related methods research for FY2012–2017, based on the exposures coded for each project. The topics are ranked based on average proportion of prevention projects coded for each exposure and the table presents the top ten; the complete table is provided in Appendix Table 2.
With 128 tests for trend, the test is significant only if p-trend<0.00039.
Table 3 summarizes the results for coding projects based on their outcomes. Cancer, infectious disease, healthcare delivery, mental health, health-related quality of life, substance use disorder, medication/device, neurologic diseases (excluding Alzheimer disease), heart disease, and tobacco completed the top ten outcome topics. Medication/device can be an outcome, for example, in a study to determine whether an intervention influences adherence to medications, the use of a medical device, or the safety of a medication. Genetics can be an outcome, for example, in a study looking at the impact of an intervention on DNA methylation. Over FY12–17, there was a decrease in studies that included a cancer outcome, from 14.0% to 5.5% (p=0.00012). None of the other outcome topics presented a significant trend over FY12–17.
Table 3.
Estimated Proportion of Projects by Topic and Fiscal Year (FY) Based on Their Outcomesa.
| Rank | Topic | FY2012 % (95% CI) |
FY2013 % (95% CI) |
FY2014 % (95% CI) |
FY2015 % (95% CI) |
FY2016 % (95% CI) |
FY2017 % (95% CI) |
Average % (95% CI) |
p-trendb |
|---|---|---|---|---|---|---|---|---|---|
| 1 | Other or unclear | 59.7 (54.1, 65.0) |
62.7 (56.7, 68.2) |
60.0 (54.3, 65.4) |
61.2 (56.0, 66.2) |
62.8 (58.1, 67.3) |
71.2 (66.7, 75.4) |
62.9 (60.8, 65.0) |
0.00395 |
| 2 | Cancer | 14.0 (10.2, 19.0) |
14.1 (10.4, 19.0) |
13.9 (10.2, 18.5) |
12.0 (9.6, 14.9) |
11.5 (9.1, 14.4) |
5.5 (3.7, 8.0) |
11.8 (10.4, 13.3) |
0.00012 |
| 3 | Infectious disease | 10.7 (7.5, 15.0) |
6.4 (5.1, 8.1) |
11.2 (8.2, 15.1) |
10.3 (7.7, 13.6) |
10.3 (7.5, 13.9) |
9.9 (7.7, 12.5) |
9.9 (8.7, 11.2) |
0.68175 |
| 4 | Healthcare delivery |
7.7 (5.8, 10.2) |
7.1 (5.0, 10.1) |
6.2 (5.0, 7.7) |
9.5 (7.4, 12.2) |
11.1 (8.7, 14.0) |
7.7 (5.7, 10.2) |
8.2 (7.3, 9.1) |
0.14265 |
| 5 | Mental health | 7.4 (5.5, 10.0) |
11.3 (7.7, 16.3) |
5.5 (3.7, 8.2) |
5.9 (4.7, 7.4) |
7.6 (5.5, 10.4) |
6.8 (4.9, 9.3) |
7.3 (6.3, 8.4) |
0.22846 |
| 6 | Health related quality of life |
5.6 (3.8, 8.2) |
7.2 (5.0, 10.1) |
8.5 (5.8, 12.3) |
4.0 (3.1, 5.2) |
7.0 (5.6, 8.5) |
7.0 (4.7, 10.2) |
6.6 (5.7, 7.6) |
0.85697 |
| 7 | Substance abuse | 6.5 (5.2, 7.9) |
6.7 (4.6, 9.5) |
5.5 (3.8, 7.9) |
7.1 (5.8, 8.7) |
5.9 (4.7, 7.4) |
5.7 (3.9, 8.2) |
6.2 (5.5, 7.0) |
0.58219 |
| 8 | Medication/device | 5.9 (3.4, 10.0) |
4.7 (2.4, 8.9) |
7.9 (5.1, 12.2) |
2.8 (2.0, 3.8) |
4.6 (2.7, 7.4) |
8.7 (5.7, 13.1) |
5.9 (4.7, 7.2) |
0.53844 |
| 9 | Neurological disease |
4.4 (2.7, 7.0) |
2.4 (1.6, 3.5) |
6.4 (3.8, 10.7) |
6.0 (3.7, 9.5) |
8.4 (5.8, 12.2) |
5.9 (3.7, 9.2) |
5.6 (4.6, 6.8) |
0.02165 |
| 10 | Heart disease | 4.9 (2.9, 8.4) |
6.4 (3.9, 10.4) |
3.6 (2.7, 4.6) |
6.4 (4.1, 9.8) |
7.8 (5.2, 11.5) |
4.1 (3.2, 5.3) |
5.4 (4.6, 6.5) |
0.80328 |
Tabled values are the estimated proportions and 95% CI of NIH projects funded as grants or cooperative agreements that supported primary and secondary prevention research in human, as well as related methods research for FY2012–2017, based on the outcomes 21 coded for each project. The topics are ranked based on average proportion of prevention projects coded for each outcome and the table presents the top ten; the complete table is provided in Appendix Table 3.
With 128 tests for trend, the test is significant only if p-trend<0.00039.
Table 4 summarizes the results for coding projects based on their population foci. Most projects did not specify a population focus or focused on the general population and were coded as “Adult/Unclear,” representing 71.5% of the projects. Other commonly selected foci included youth, urban, older adults, and pregnant or post-partum women. None of the population foci topics presented a significant trend over FY12–17.
Table 4.
Estimated Proportion of Projects by Topic and Fiscal Year (FY) Based on Their Population Focia.
| Topic | FY2012 % (95% CI) |
FY2013 % (95% CI) |
FY2014 % (95% CI) |
FY2015 % (95% CI) |
FY2016 % (95% CI) |
FY2017 % (95% CI) |
Average % (95% CI) |
p-trendb |
|---|---|---|---|---|---|---|---|---|
| Adult/unclear | 71.2 (66.7, 75.3) |
73.0 (68.9, 76.8) |
74.6 (70.3, 78.5) |
70.1 (65.9, 74.0) |
71.8 (68.4, 74.9) |
68.5 (63.9, 72.8) |
71.5 (69.8, 73.2) |
0.20969 |
| Youth (infants, children, adolescents) |
25.2 (21.3, 29.6) |
25.7 (22.0, 29.8) |
21.3 (18.0, 25.0) |
26.5 (22.8, 30.6) |
24.6 (21.4, 28.0) |
29.2 (24.9, 33.9) |
25.4 (23.8, 27.0) |
0.18494 |
| Older adults/elderly | 7.4 (5.4, 10.0) |
9.2 (7.5, 11.2) |
8.9 (6.0, 13.0) |
7.4 (5.2, 10.3) |
11.5 (8.7, 15.1) |
11.9 (8.7, 16.1) |
9.4 (8.2, 10.6) |
0.02378 |
| Urban | 9.5 (8.0, 11.4) |
9.2 (7.6, 11.2) |
8.5 (6.8, 10.5) |
10.3 (8.6, 12.2) |
10.7 (9.0, 12.7) |
7.3 (6.0, 8.9) |
9.2 (8.5, 10.0) |
0.43888 |
| Pregnant and/or post- partum women |
6.0 (4.2, 8.4) |
7.4 (5.2, 10.3) |
6.0 (4.3, 8.4) |
6.9 (5.0, 9.4) |
7.7 (5.6, 10.6) |
6.7 (8.2, 5.4) |
6.7 (5.9, 7.7) |
0.51962 |
| Low income | 3.9 (3.0, 5.1) |
3.8 (2.8, 5.1) |
3.5 (2.7, 4.6) |
3.9 (3.0, 5.1) |
5.0 (3.9, 6.4) |
5.5 (3.7, 8.0) |
4.3 (3.7, 4.8) |
0.06694 |
| LGBTI | 2.0 (1.4, 2.9) |
2.6 (1.8, 3.7) |
3.3 (1.8, 6.0) |
2.4 (1.7, 3.4) |
3.0 (2.1, 4.1) |
2.5 (1.8, 3.5) |
2.6 (2.2, 3.2) |
0.46184 |
| People with disabilities | 2.1 (1.5, 3.0) |
2.6 (1.1, 6.1) |
3.0 (1.6, 5.5) |
2.3 (1.6, 3.3) |
1.7 (1.2, 2.6) |
3.2 (1.7, 5.7) |
2.5 (2.0, 3.2) |
0.73181 |
| Rural | 2.4 (1.7, 3.3) |
2.2 (1.5, 3.1) |
1.8 (1.3, 2.6) |
2.0 (1.4, 2.8) |
2.4 (1.7, 3.4) |
2.9 (2.2, 4.0) |
2.3 (2.0, 2.6) |
0.29727 |
| Military/veterans | 0.9 (0.5, 1.6) |
0.9 (0.5, 1.5) |
2.0 (0.8, 4.9) |
0.7 (0.4, 1.3) |
1.7 (1.1, 2.5) |
0.8 (0.4, 1.4) |
1.2 (0.8, 1.6) |
0.94862 |
| Incarcerated/ institutionalized |
0.7 (0.4, 1.3) |
1.8 (0.5, 6.0)c |
0.6 (0.3, 1.0) |
0.4 (0.2, 1.0)d |
0.8 (0.5, 1.5) |
1.2 (0.8, 1.9) |
0.9 (0.6, 1.4) |
0.98876 |
Tabled values are the estimated proportions and 95% CI of NIH projects funded as grants or cooperative agreements that supported primary and secondary prevention research in human, as well as related methods research for FY2012–2017, based on the population foci coded for each project. The topics are ranked based on average proportion of prevention projects coded for each population focus.
With 128 tests for trend, the test is significant only if p-trend<0.00039.
Based on a very small sample.
LGBTI, lesbian, gay, bisexual, transgender, intersex.
Table 5 summarizes the results for coding projects based on their research design. Observational studies were included in 63.3% of projects, whereas an analysis of existing data was included in 43.4%, and methods research was included in 23.9%. Only 18.2% of prevention projects included a randomized intervention. None of the research design topics presented a significant trend over FY12–17.
Table 5.
Estimated Proportion of Projects by Topic and Fiscal Year (FY) Based on Their Study Designsa.
| Topic | FY2012 % (95% CI) |
FY2013 % (95% CI) |
FY2014 % (95% CI) |
FY2015 % (95% CI) |
FY2016 % (95% CI) |
FY2017 % (95% CI) |
Average % (95% CI) |
p-trendb |
|---|---|---|---|---|---|---|---|---|
| Observational study | 60.1 (54.7, 65.2) |
66.8 (61.3, 71.9) |
60.9 (55.4, 66.2) |
66.9 (62.1, 71.3) |
63.9 (59.7, 67.8) |
62.0 (56.5, 67.1) |
63.3 (61.2, 65.4) |
0.72963 |
| Analysis of existing data | 36.4 (31.8, 41.3) |
45.4 (39.8, 51.1) |
40.0 (35.1, 45.1) |
46.1 (41.1, 51.3) |
50.7 (46.1, 55.4) |
43.2 (38.2, 48.3) |
43.4 (41.4, 45.5) |
0.01015 |
| Methods research | 27.3 (22.4, 32.7) |
22.9 (18.2, 28.2) |
24.6 (20.1, 29.7) |
25.4 (20.7, 30.8) |
23.1 (19.2, 27.6) |
20.3 (16.1, 25.3) |
23.9 (22.0, 26.0) |
0.09598 |
| Randomized intervention study |
20.3 (16.5, 24.8) |
15.9 (12.7, 19.7) |
18.8 (15.0, 23.3) |
16.3 (13.8, 19.3) |
20.5 (17.6, 23.8) |
17.0 (14.0, 20.4) |
18.2 (16.7, 19.7) |
0.63870 |
| Pilot/feasibility/proof-of- concept/safety |
11.9 (8.9, 15.8) |
8.6 (6.3, 11.7) |
14.3 (10.8, 18.6) |
10.6 (8.0, 14.0) |
10.2 (7.8, 13.2) |
11.5 (8.1, 16.2) |
11.3 (10.0, 12.8) |
0.90201 |
| Non-randomized intervention study |
5.3 (4.2, 6.7) |
5.1 (3.9, 6.6) |
8.3 (5.4, 12.4) |
5.0 (3.9, 6.4) |
8.2 (6.0, 11.0) |
5.4 (3.6, 7.9) |
6.2 (5.4, 7.2) |
0.49871 |
| Other or unclear | 2.7 (1.3, 5.4) |
3.4 (2.4, 4.7) |
5.0 (3.0, 8.2) |
4.8 (2.8, 8.2) |
2.4 (1.7, 3.5) |
2.3 (1.1, 5.0) |
3.5 (2.8, 4.4) |
0.50368 |
Tabled values are the estimated proportions and 95% CI of NIH projects funded as grants or cooperative agreements that supported primary and secondary prevention research in human, as well as related methods research for FY2012–2017, based on the study designs coded for each project. The topics are ranked based on average proportion of prevention projects coded for each study design.
With 128 tests for trend, the test is significant only if p-trend<0.00039.
Table 6 summarizes the results for coding projects based on their type of prevention research. Primary prevention was the most common type of prevention research, included in 66.1% of projects. Methods research and secondary prevention were also common, included in 23.3% and 19.2% of projects, respectively. None of the type of prevention research topics presented a significant trend over FY12–17.
Table 6.
Estimated Proportion of Projects by Topic and Fiscal Year (FY) Based on Their Type of Prevention Researcha.
| Topic | FY2012 % (95% CI) |
FY2013 % (95% CI) |
FY2014 % (95% CI) |
FY2015 % (95% CI) |
FY2016 % (95% CI) |
FY2017 % (95% CI) |
Average % (95% CI) |
p-trendb |
|---|---|---|---|---|---|---|---|---|
| Primary preventionc | 62.3 (56.8, 7.6) |
70.9 (65.4, 75.8) |
63.8 (58.2, 69.1) |
67.8 (62.5, 72.6) |
64.2 (59.4, 68.7) |
68.3 (62.8, 73.4) |
66.1 (64.0, 68.2) |
0.46362 |
| Methods research | 26.2 (21.4, 31.7) |
22.0 (17.4, 27.4) |
23.6 (19.2, 28.7) |
24.9 (20.2, 30.3) |
22.8 (18.9, 27.2) |
19.9 (15.7, 24.9) |
23.3 (21.3, 25.3) |
0.16147 |
| Preventing progression or recurrenced,e |
18.1 (14.8, 21.9) |
18.5 (14.8, 22.8) |
22.5 (17.9, 27.7) |
15.0 (12.4, 18.0) |
21.3 (17.7, 25.3) |
19.7 (15.8, 24.4) |
19.2 (17.6, 20.9) |
0.62051 |
| Screening for early diseasee |
3.8 (2.9, 5.0) |
5.3 (4.1, 6.8) |
4.8 (3.7, 6.0) |
5.9 (4.7, 7.4) |
5.6 (4.5, 7.1) |
5.4 (3.8, 7.8) |
5.1 (4.6, 5.7) |
0.10848 |
| Screening for risk factorsc |
1.0 (0.6, 1.7) |
0.5 (0.2, 1.0)f |
1.1 (0.6, 1.8) |
0.5 (0.2, 1.0)f |
0.3 (0.1, 0.9)f |
0.6 (0.3, 1.1)e |
0.7 (0.5, 0.9) |
0.08744 |
Tabled values are the estimated proportions and 95% CI of NIH projects funded as grants or cooperative agreements that supported primary and secondary prevention research in human, as well as related methods research for FY2012–2017, based on the type of prevention research coded for each project. The topics are ranked based on average proportion of prevention projects coded for each type of prevention research.
With 128 tests for trend, the test is significant only if p-trend<0.00039.
Primary prevention includes preventing a new health condition, promoting health in the general population, or identifying risk factors for a new health condition.
Preventing progression or recurrence includes preventing progression of a health condition, preventing recurrence in those with a known health condition, or identifying risk factors for progression or recurrence.
Secondary prevention includes preventing progression or recurrence, screening for early disease, and screening for risk factors.
Based on a very small sample.
DISCUSSION
Current Findings
During FY12–17, ODP estimates that 16.7% of projects and 22.6% of dollars awarded by NIH through research grants or cooperative agreements were used to support primary and secondary prevention research in humans, together with methods studies to support that research. A question for NIH and for the research community is whether these proportions are appropriate. There is no agreed upon answer to this question, and the need for prevention research in humans is variable depending on the stage of research for a given area (e.g., mechanistic research, development of measures, identification of risk factors, intervention development). NIH must also balance the needs of many areas of science as it seeks to fulfill its mission of seeking fundamental knowledge about the nature and behavior of living systems and applying that knowledge to enhance health, lengthen life, and reduce illness and disability. Advancing opportunities in biomedical research in health promotion and disease prevention is one of three key targets in the NIH-Wide Strategic Plan (www.nih.gov/news-events/news-releases/nih-unveils-fy2016–2020-strategic-plan) and the data presented here will inform the conversation about the appropriate level of support for prevention research.
A recent report from the Global Burden of Disease project indicates that 74% of the county-level variability in life expectancy in the can be attributed to well-established risk factors for disease, that most of the association between socioeconomic and race/ethnicity factors and life expectancy is mediated through those risk factors, and that the addition of healthcare factors does not account for additional variation.6 This finding, building on earlier studies also pointing to the important role of well-established risk factors across diseases,7 presents a strong case for directing a meaningful fraction of the nation’s biomedical research and healthcare resources towards addressing those risk factors. As many chronic diseases have the same or similar risk factors, the research community should keep that in mind as it considers whether the fraction of NIH resources devoted to prevention research is appropriate.
The topics that were selected for rationale by more than 5% of the projects in the prevention portfolio included many of the leading causes of death and established risk factors. Other risk factors that might be expected were not eligible (e.g., diet/nutrition, physical activity), because rationale topics were limited to health conditions with only five exceptions.e The rank order of the topics reflects the funding decisions of the ICs, which in turn reflect the scientific merit of the applications and the programmatic priorities of the ICs. Importantly, the rank ordering does not reflect any a priori agreement among or within the ICs to make their allocations according to this or any other schema, and the ordering would vary considerably if presented by the ICs.
The exposures selected for prevention projects were not widely distributed across the 41 topics that were included in the taxonomy. The results show 57.3% of the abstracts identified at least one exposure as other/unclear, and only four topics were selected for at least 5% of the prevention portfolio. This highlights a limitation of this approach, and so additional topics within the ODP exposure category may be required going forward.
The outcomes reported for prevention projects were widely distributed, reflecting the wide range of health conditions and health behaviors of interest to prevention researchers supported by NIH. Notably, the single most preventable cause of death and disability, tobacco, was selected for only 5.1% of the prevention research portfolio, and many of the other leading risk factors were selected for less than 5% of the portfolio (e.g., alcohol, obesity, diet/nutrition, physical activity).
The vast majority of projects did not point to any specific group as a population focus and instead addressed the general adult population. Many included some of the population foci, but did not identify them as a special target.
The high proportion of studies that included an observational study (63.3%), an analysis of existing data (43.4%), or methods research (23.9%), compared with the significantly lower proportion that included a randomized intervention (18.2%), was surprising. Because so much is already known about the risk and protective factors that explain a large fraction of the variation in death, or life expectancy,6 across diseases, the nation would be well served by having more of NIH’s prevention resources focused on applying that knowledge to the development and testing of interventions to prevent the leading causes of death by addressing those risk and protective factors.
Primary prevention was by far the most common type of prevention research in the portfolio, at 62.3%, whereas methods research and secondary prevention represented smaller, but still substantial fractions, at 26.2% and 18.1%. Studies focused on screening were much less common, particularly screening for risk factors. This finding is consistent with reports from the U.S. Preventive Services Task Force which regularly call for more evidence on screening for preventable conditions, particularly related to screening for cancer.8
Comparison with Previous Studies
Calitz et al.1 published a report in the American Journal of Preventive Medicine in 2015 on NIH funding for behavioral interventions to prevent chronic diseases. Some of their results were very different from the results presented here; for example, they reported that 21% and 70% of prevention awards were primary and secondary prevention, respectively, where this paper reports essentially the opposite. Other results were similar; for example, they reported that 7% and 4% of prevention awards were for tobacco and alcohol, respectively, whereas this paper reports 5% for each as an outcome. The two studies used very different methods, so it is not surprising that the results were often different. They drew their data only from the NIH Research Portfolio Online Reporting Tools for Expenditures and Results Tool (RePORTER), a public-facing research portfolio online reporting toolf; this study drew data from NIH’s internal grant database, IMPAC II, which is more complete. They selected projects from FY2010 to FY2012; this study drew data from FY12 to FY17. They drew data from only eight of the 27 NIH ICs; this study included all 27. They coded awards identified by RePORTER as human subjects prevention studies; this study selected projects using ML tools. They included all activity codes; this study was restricted to 11 activity codes that represented most of the NIH research projects supported by grants and contracts.
Limitations
The authors made no attempt to ascertain the proportion of a project’s proposed activities that was focused on a particular topic. Instead, if the text of the abstract met the criteria for a topic, that topic was selected for that project. To illustrate, if an abstract had four specific aims, and at least one of them qualified as prevention research under ODP’s definition, that project was coded as a prevention project, even if many of the activities described in the abstract did not qualify as prevention research. The authors understood from the outset that this would mean that the results might overstate the level of prevention research, or the level of any other topic, but judged that parsing the proportion of a project that was focused on a particular topic based on the abstract alone was impossible. As a result, the figures reported in Table 1 for the proportion of the total number of projects and dollars for projects that were coded as prevention (16.7% and 22.6% respectively) should be viewed as an upper bound rather than as exact estimates. Fortunately, this limitation is less likely to affect the rank ordering of topics within categories.
Another limitation is that the portfolio analysis focused exclusively on the extramural research program at NIH, and within that program, on grants and cooperative agreements. These activity codes encompass most of the primary and secondary prevention research in humans supported by NIH, but not all of it. Some Institutes use both contracts and their intramural research program to support prevention research, including some of the important vaccine research supported by NIH. The intramural research program represented 10.9% of the FY17 NIH budget, and contracts represented 5.9%. However, because only a fraction of the funds used to support intramural research and contracts is used for primary and secondary prevention research in humans, excluding these projects is unlikely to have biased the characterization of the NIH prevention research portfolio reported here.
Another limitation is that the taxonomy included only 128 topics and it is not possible to report on any topics that were not included. The data presented here suggest that there were missing topics for exposure, but that may also be true for other categories.
NEXT STEPS
This paper provides the most detailed and carefully validated analysis of NIH’s prevention portfolio ever conducted. ODP looks forward to a robust discussion with colleagues at NIH and with the prevention research community regarding the priorities suggested by these findings. One obvious discussion point is whether NIH is investing too little in prevention research generally, and in preventive intervention trials in particular. Another is whether the observed rank ordering of rationale and outcome topics is appropriate, and whether they align sufficiently with the leading and actual causes of death and disability in the U.S.
From the outset, ODP recognized that manual coding would be impractical for the entire NIH portfolio. Factors, such as human error, may lead to inconsistencies in coding across coders and time. In collaboration with the NIH Office of Portfolio Analysis, ODP will use the ML tools to evaluate the sensitivity and specificity for each of the topics included in the taxonomy. ODP will continue to apply its methods as new grants are funded. ODP will also explore the utility of examining submitted applications, not just funded projects.
Supplementary Material
ACKNOWLEDGMENTS
All authors were employees of the NIH Office of Disease Prevention when they were actively engaged in work related to this paper. NIH is the sole source of support for the work reported.
The authors would like to thank the team of research analysts and developers at IQ Solutions for their work on coding, Luis Ganoza for his help preparing the tables, and Kat Schwartz and David Tilley for their thoughtful comments on this manuscript.
Footnotes
Primary prevention includes preventing a new health condition, promoting health in the general population, or identifying risk factors for a new health condition.
Secondary prevention includes preventing progression of disease, preventing recurrence in those with a known health condition, and identifying risk factors for progression or recurrence.
Activity codes were: R01, R03, R21, R43, R44, R56, P01, P50, U01, U19, U54, and UM1 (https://grants.nih.gov/grants/funding/ac_search_results.htm).
R01 projects were used as the training set because the R01 is the oldest and most common grant at NIH. The R01 provides support for discrete, health-related research projects that are relevant to the mission of the NIH.
A higher proportion of projects was selected for quality control at the beginning of the coding process for each activity code, and in weeks when the quality control results indicated a Κ <0.70.
No financial disclosures were reported by the authors of this paper.
Publisher's Disclaimer: This is a PDF file of an unedited manuscript that has been accepted for publication. As a service to our customers we are providing this early version of the manuscript. The manuscript will undergo copyediting, typesetting, and review of the resulting proof before it is published in its final citable form. Please note that during the production process errors may be discovered which could affect the content, and all legal disclaimers that apply to the journal pertain.
References
- 1.Calitz C, Pollack KM, Millard C, Yach D. National Institutes of Health funding for behavioral interventions to prevent chronic diseases. Am J Prev Med 2015;48(4):462–471. https://doi.org/10.1016/j.amepre.2014.10.015. [DOI] [PubMed] [Google Scholar]
- 2.Murray DM, Cross WP, Simons-Morton D, et al. Enhancing the quality of prevention research supported by the National Institutes of Health. Am J Public Health 2015;105(1):9–12. https://doi.org/10.2105/AJPH.2014.302057. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 3.Mokdad AH, Ballestros K, Echko M, et al. The state of U.S. health, 1990–2016. JAMA 2018;319(14):1444–1472. https://doi.org/10.1001/jama.2018.0158. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 4.Villani J, Schully SD, Meyer P, et al. A Machine Learning Approach to Identify NIH-Funded Applied Prevention Research. Am J Prev Med In press 2018. [DOI] [PMC free article] [PubMed]
- 5.Wojcik BE, Stein CR, Devore RB, Hassell H. The challenge of mapping between two medical coding systems. Mil Med 2006;171(11):1128–1136. https://doi.org/10.7205/MILMED.171.11.1128. [DOI] [PubMed] [Google Scholar]
- 6.Dwyer-Lindgren L, Bertozzi-Villa A, Stubbs RW, et al. Inequalities in life expectancy among U.S. counties, 1980 to 2014: temporal trends and key drivers. JAMA Intern Med 2017;177(7):1003–1011. https://doi.org/10.1001/jamainternmed.2017.0918. [DOI] [PMC free article] [PubMed] [Google Scholar]
- 7.Mokdad AH, Marks JS, Stroup DF, Gerberding JL. Actual causes of death in the United States, 2000. JAMA 2004;291(10):1238–1245. https://doi.org/10.1001/jama.291.10.1238. [DOI] [PubMed] [Google Scholar]
- 8.U.S. Preventive Services Task Force. Sixth Annual Report to Congress on High-Priority Evidence Gaps for Clinical Preventive Services Published 2016. [PubMed]
Associated Data
This section collects any data citations, data availability statements, or supplementary materials included in this article.