Training and Career Development for Comparative Effectiveness Research Workforce Development: CTSA Consortium Strategic Goal Committee on Comparative Effectiveness Research Workgroup on Workforce Development

Abstract

As comparative effectiveness research (CER) increasingly becomes a national priority, there are increased needs for training and workforce development in this area. Particularly important CER competencies include clinical epidemiology, biomedical informatics, economic analyses, systematic reviews, clinical practice guideline development, use of large databases and electronic health records for research, practice‐based research, implementation and dissemination, health services research, and decision analysis. Institutions funded by Clinical and Translational Science Awards (CTSAs) should be one of the central stakeholders in providing training and career development in CER. Survey results regarding the current CER capacity and needs of CTSAs are presented, and recommendations are provided. Volume 5: 258–262.

In galvanizing the new US research priority in comparative effectiveness research (CER), it is paramount that there be a sufficient workforce with the requisite expertise and skill set. In 2009, there were a number of reports (some published and others in draft form) by the Institute of Medicine, ¹ , ² , ³ , ⁴ Clinical and Translational Science Award (CTSA) workgroups, ⁵ , ⁶ and other federal committees, ⁷ which speak about CER priorities and, to varying degrees, CER workforce gaps, training needs, and competencies. A white paper, summarizing the CTSA National Consortium activities in this area, has been recently published. ⁸ While these reports speak to CER in general, our briefer report focuses specifically on CTSA programs, and how they might foster education and career development for a CER workforce. In particular, the charge for our workgroup was threefold:

• 1To examine the workforce needs and gaps in CER.
• 2To begin to identify the requisite competencies for CER.
• 3To identify potential mechanisms to support training in CER, particularly within CTSAs.

Workforce Needs and Gaps

CTSA survey

In 2009, the CTSA CER Committee Needs/Capacity Workgroup Strategic conducted a survey of the 38 funded CTSA programs with a good (n= 33) response rate. Their findings are summarized in “Assessment of Comparative Effectiveness Research Capacity and Needs for CTSA Institutions.” ⁵ , ⁸ We have excerpted from the full report responses to two questions: first, on the CER research conducted in the CTSA respondent’s institution in the past 3 years, and a second, on the degree to which responding institutions feel their capacity for CER research needs to increase. Table 1 summarizes the proportion of institutions that reported they had “no to minimal research” in CER and the proportion that felt their institution needed to increase CER capacity “to a large extent.” Priority areas identified by the CTSAs include electronic health record research, registry‐based research, economic analyses, practice‐based network research, practical clinical trials, observational cohort studies, systematic reviews/meta‐analyses, and research that incorporate stakeholders and patient preferences.

Table 1.

Summary of 2009 survey of 33 CTSA institutions on CER capacity and needs.

Research area/competency	No to minimal research	Need large growth
Electronic health record (EHR) research	58%	53%
Registry (clinically based) research	52%	50%
Economic analyses	46%	34%
Working with other research networks (HMO, CTRN, CCOP)	49%	25%
Stakeholder involvement in planning and implementing research	55%	19%
Practice‐based research network research	39%	31%
Analysis of approaches for delivering and/or paying for care	46%	23%
Clinical trials (especially effectiveness)	30%	38%
Cohort studies, long‐term observational	33%	34%
Systematic reviews/meta‐analysis	42%	19%
Patient preference assessment for care/outcomes	43%	16%
Decision analysis	46%	13%
Knowledge implementation/transfer research	31%	22%
Outcomes analysis outside institution	33%	19%
Development of outcome/quality measures	27%	19%
Novel methods development for data analysis or modeling	15%	9%

As the table shows, the CTSAs have some capacity to conduct CER, although there is a need for additional growth in many areas. The implication from this survey is that many CTSAs can begin conducting CER, and with additional resources and emphasis, additional capacity can be built. Since CTSAs are major institutional programs, they are most likely to succeed in bringing together institutional resources to move forward the CER agenda.

Institute of medicine (IOM) roundtable

The one CER report, which addresses in greatest detail both competencies and workforce needs, is the IOM Roundtable. ⁴ Several parts of this report are excerpted briefly below, namely, seven key questions, examples of projected workforce needs, and conclusions.

Key Questions for Addressing CER Workforce Needs

The report notes: “The first challenge in defining the CER workforce is to grapple with the larger question of the quantity of CER we deem necessary for the learning health care system. In order to quantify the needs, we must answer these questions”:

• 1What quantity of comparative clinical trials and other clinical research will be required?
• 2What quantity of CER systematic reviews will need to be performed?
• 3What amount of pharmacoepidemiological and related analysis will be required, or even possible, especially, given the small number of pharmacoepidemiologists?
• 4How many medical centers will be willing or able to engage in using their electronic health record systems and/or local guideline implementation to provide data for CER?
• 5What quantity of clinical practice guidelines (CPGs) will need to be produced?
• 6What types and amounts of health services research (HSR) will be necessary for CER?
• 7What types and quantities of dissemination will be required for CER? At how many levels will the content require reformatting?

Examples of Projected Workforce Needs

Although the IOM Roundtable report tried to project CER workforce needs in greater detail than other CER reports, it was still only able to give examples for selected CER disciplines, and the scope of projections varied widely, ranging from broad estimates for biomedical informatics (entire nation) to more focused estimates for pharmacoepidemiology (only needed by FDA).

• 1Evidence reports
  • a. The Cochrane Collaboration originally estimated a need for 20,000 reviews; to date, they have completed 3,539 and another 1,868 are proposed or underway.
  • b. Agency for Healthcare Research and Quality (AHRQ) Evidence‐based Practice Centers have produced 168 evidence reports and 16 technical reviews.
  • c. Drug Effectiveness Review Project produced 28 original reports and updated 45 reports in its first 3 years.
  • d. Updating systematic reviews published in the medical literature have a half‐life of about 5.5 years, with about 23% requiring updating within 2 years of publication.
• 2Pharmacoepidemiology: the FDA safety program alone may need 80–100 more pharmacoepidemiologists.
• 3Biomedical informatics
  • a. Estimates from the US as well as the United Kingdom suggest a need for one information technology (IT) full‐time equivalent worker per 56 non‐IT workers in health care delivery workforce.
  • b. On the basis of the above and other projections, 40,000 new IT full‐time equivalent workers might be needed.
  • c. However, it is not likely, all or most of the above are CER‐specific needs. Many may be IT needs related to health care delivery or other services.
• 4CPGs. AHRQ currently has 2000 CPGs that, if updated every 5 years, would require 400 review teams to be assembled per year, even assuming zero growth in new CPGs.

There are factors that could increase CER workforce needs, such as job market competition for CER skills, especially in biomedical informatics (healthcare, industry, and public and private sector needs), pharmacoepidemiology (FDA), and clinical practice (e.g., primary care physicians who might do HSR go into practice). There are also factors that could decrease CER workforce needs in the US, such as international collaborations (e.g., National Institute for Clinical Excellence guideline development groups in the United Kingdom, and comparable groups in Canada and Sweden; Cochrane collaborations).

Conclusions

The IOM roundtable refrained from any quantitative projections. Because their rationale is both well‐articulated and salient to our report, it is cited at length in the three paragraphs below. ⁴

“We originally intended to provide a quantitative first approximation of workforce needs for CER. However, as the authors developed the framework and explored the issues more deeply, it was apparent that there are too many unanswered questions about the scope, breadth, and quantity of CER that needs to be clarified to achieve larger goals for a learning health care system. This view was validated by many of the experts listed in the Acknowledgements, who advised against attempting to quantify needs with such an unclear picture and future for CER.

There are a number of reasons why a quantitative assessment of the CER workforce is not possible. The main one is that we do not know the true scope of CER. For example, in the area of clinical epidemiology, where there is probably the most clarity about needs in any of the areas we assessed, the numbers of systematic reviews, practical and other clinical trials, and pharmacoepidemiological analyses required are not clear. While we have a good handle on the personnel required for systematic reviews, the requirements for the other categories of clinical trials and pharmacoepidemiological analyses are much less clear.

Beyond clinical epidemiology, the picture becomes even less certain. While biomedical informatics, development and implementation of clinical guidelines, and dissemination could become a major part of CER, the amount of each that needs to be done, or that falls under the rubric of CER, is not clear. Furthermore, in all of these areas, CER would be a secondary activity to the larger task of maintaining IT systems for clinical care; using guidelines to improve the quality, safety, consistency, and cost‐effectiveness of operational clinical care; and disseminating all types of clinical knowledge. How much of the work would actually be CER is not known or easy to determine. Even in HSR, the amount of research to be done that could be classified as CER is not certain. Clearly, in an analysis held under the rubric of “evidence‐based medicine,” there is little evidence to make sound judgments about specific workforce needs.”

Other CER reports

Other CER reports call for an increased workforce but are less specific and detailed than either the CTSA survey or the IOM Roundtable.

Institute of Medicine Main CER Report

In the main IOM CER report, the committee concludes: “The size of the qualified CER workforce is not known and the workforce needed to perform CER must be defined, assessed, and trained. NIH, FDA, AHRQ’s DEcIDE Research Network, CERTs, CMS, VA, and CMTP are among the organizations focused on the development of new evidence from well‐designed comparative clinical trials and observational studies… Gauging the capacity of the current CER workforce is difficult because so many disciplines are involved and so many educational pathways to the field exist… Nonetheless, ARRA’s infusion of $1.1 billion into CER will clearly stress the limits of the current CER workforce. ARRA appropriations increased AHRQ’s CER budget tenfold. Aggregate current NIH spending on CER is not known, but the Institutes will receive at least an additional $400 million to conduct CER.’’

Federal Coordinating Council Report

Regarding workforce, the Federal Coordinating Council simply states: “Greater investment is needed in developing education and training programs to support the development of professional talent, the development of methods for linking and using databases for CER, the development of new methodologies for pragmatic trials, effective translation and adoption of CER findings into practice, modeling approaches for CER, and evaluation of the impact of CER.”

Competencies

Identifying potential competencies in any given area and coming to a consensus requires the appropriate stakeholders coupled with a deliberative and iterative group process. In collaboration with the NCRR, the CTSA Education and Career Development Work facilitated such a process over 18 months (January 2008–June 2009) and identified 14 areas with specified competencies relevant to clinical and translational research (CTR). ⁷ Notably, CER competencies are not explicitly included in the original CTSA report, although many of the themes may include specific competencies that are relevant to CER. Four of the CER reports by other groups have identified a number of competencies relevant to CER that are summarized in Table 2 .

Table 2.

CER curriculum and/or competencies identified by four expert groups.

Curriculum/competencies (reference)	IOM roundtable4	CTSA CER 4–5a group5	CTSA CER 4–5b group6	IOM CER report1
Clinical epidemiology	X		X	X
• Practical clinical trials (pragmatic; effectiveness)	X	X	X	X
• Nontrial clinical research	X			X
• Systematic reviews (including meta‐analyses and technology assessment)	X	X	X	X
• Pharmacoepidemiology	X
• Clinical epidemiology methods research	X			X
Biomedical informatics	X	X	X	X
• Electronic health records (EHR)	X
• Reuse of clinical data	X
• Information needs assessment	X
• Data mining/other knowledge discovery	X
• Ontology development/knowledge management	X
• Computer sciences			X
Clinical Practice Guidelines (CPGs)	X
Health services research	X	X		X
Implementation and dissemination	X		X
Biostatistics (in CER)		X	X	X
Large database research			X	X
Health economics/cost‐effectiveness		X	X	X
Decision analysis/cognitive sciences			X	X
Practice‐based network research			X
Community‐engaged research			X
Library sciences				X
Communication (of research findings)				X
Clinical medicine				X
Genomics				X

Key points

• 1The Competency Development Workgroup of the CTSA Education/Career Development Key Function Committee listed generic competencies for CTR, most of which are relevant to CER as a specific type of CTR. ⁷ The 14 competencies include defining the research question, critiquing the literature, study design, research implementation, sources of error, biostatistics, biomedical informatics, responsible conduct of research, scientific communication, cultural diversity, translational teamwork, leadership, cross‐disciplinary research, and community engagement.
• 2Of these, there are some competencies with special issues particularly salient to CER including research ethics (e.g., complex issues related to community‐engaged research), biostatistics, pragmatic clinical trials, biomedical informatics, etc.
• 3As shown in Tables 1 and 2 , there are CER core competencies not clearly covered in generic competencies, such as CPGs, electronic health records research, practice‐based network research, and implementation/dissemination.
• 4One area closely aligned with CER is community‐engaged research. CER focuses primarily on clinical samples (especially comparing treatments or tests), while community research engages community stakeholders in defining the research agenda and even projects. The CTSA Community Engagement Educational Competencies group ⁷ defined five competencies of community‐engaged research.
  • a. Examine the characteristics that bind people together as a community, including social ties, common perspectives or interests, and geography.
  • b. Appraise the role of community engagement as a strategy for identifying community health issues, translating research to communities, and reducing health disparities.
  • c. Summarize the principles and practices of the spectrum of community‐engaged research.
  • d. Analyze the ethical complexities of conducting community‐engaged research.
  • e. Specify how cultural and linguistic competence and health literacy have an impact on the conduct of community‐engaged research.
• 5Some areas overlap considerably with CER. One example is HSR, and the boundaries between CER and HSR could be more clearly defined.

Mechanisms to Support CER Training

Education, training and career development is one of the major cores of the CTSAs. This core function generally includes research training (previously funded as a K30 clinical research curriculum award), a T component (TL1, which provides stipends and tuition for trainees), and a K component (KL2, which provides funds for research and salary support for junior investigators). Through Education and Career Development Core, CTSAs are contributing to an expansion of clinical and translational research workforce and expanding the pipeline of investigators. Although some CTSAs have significant training and career development programs in CER, much of the KL2 and TL1 awards in the CTSAs are focused on more classical clinical and translational research workforce development, and not on CER.

The competencies relevant to be a skilled researcher in CER are clearly different from those needed for the typical T1 researcher. However, the full‐spectrum (T1–T2–T3–T4) of clinical and translational research programs includes HSR and CER expertise as important training and educational areas. Thus, there is an opportunity to reorient some of the CTSA functions to serve as a nidus for training the CER workforce. Expansion of currently ongoing programs and new initiatives through the CTSAs, and perhaps with attention to those CTSAs that already have AHRQ HSR training programs, would seem to be an effective strategy for the rapid increase in education and career development, via institutional T and K awards, directed at developing a high‐level workforce for CER. CTSAs could address the increase in CER workforce through K and T awards as follows.

K awards

CTSAs have a career development component (KL2) as a required program. The size of the program varies and can be as large as 25–30 positions. These institutional awards are for 2–5 years and have typical requirements for all K awards including training, protected time (75%), mentored research, design and statistical support, and availability of CTSA resources. Some of these positions could be used for CER projects either by specifically requesting CER as an area of emphasis by a CTSA or encouraging applications in this area of research. This area could be promoted by the national CTSA consortium or the National Center for Research Resources for a possible area of career development.

The training component of K awards may not formally exist at many CTSAs. However, there is no unique methodologic discipline related to the field of comparative effectiveness. Basic training could include biostatistics, epidemiology, economics, outcomes, and HSR, all of which contribute important perspectives to the translation of research to practice and policy. Ethics too will be important as institutional review boards come to review the protocols and consent forms that CER will require. Additional methodologic expertise, such as in systematic reviews/meta‐analysis, informatics, data mining, and mathematical modeling may be available at many CTSAs or could be obtained through consortia arrangement within the CTSAs or national conferences and meetings (such as the Society of Medical Decision Making).

Multidisciplinary mentoring is common at all CTSAs. Faculty from multiple relevant disciplines could come together on CER research projects within an institution. Additional consulting or distant mentoring may add special expertise from other institutions. National consulting from experts in a field or distant mentoring is frequently proposed in many independent K awards (such as K23s), so this would not be unusual for KL2 applications.

Finally, there is a great need to include multiple disciplines in the CER workforce development because of the complexity of this type of research and health care delivery. The training should encourage individuals from diverse backgrounds, such as doctoral‐level clinician‐scientists (MD, DDS, nursing, PharmD, DOT, DPT, and OD); PhD clinician‐scientists (psychology, nutrition, and others) and PhD nonclinician scientists (biostatisticians, epidemiologists, economists, social scientists, and others).

T awards

Many, if not all CTSAs, have a T component (TL1) that is either short‐term 2‐ to 3‐month training or 1–2 years of training. As noted above under K awards, training programs could be structured using the CTSA degree granting programs that include training in fundamentals of clinical research. This can be supplemented with areas, such as systematic review and meta‐analysis, large database analysis, informatics, and mathematical modeling. Issues with research mentoring are similar to career development awards. In the CTSA institutions, which also have AHRQ T32 training grants, access to some of HSR courses and curricula could provide additional avenue of training in CER. All attempts should be made to support individuals with diverse backgrounds in the training programs.

Recommendations

• 1The CTSA Education and Career Development Key Function Committee should appoint a CER Competency Workgroup to review potential competencies ( Tables 1 and 2 in this report) and provide consensus recommendations.
• 2Curricula should be developed for high‐priority CER competencies that would be available on a national (rather than just institutional) level.
• 3CER researchers from a broad range of disciplines (clinicians and nonclinician scientists) should be eligible for CTSA‐funded K and T awards.
• 4A national network of CER researchers, able and willing to provide distance mentorship, should be organized.
• 5CTSAs should partner with other federally funded CER research centers (e.g., AHRQ Evidence‐Based Practice Centers, VA‐funded Centers of Excellence) at an institutional‐ and consortium‐level to foster CER training and career development.
• 6CTSAs should encourage and include CER researchers in the K and T programs. Should additional CTSA funding be available, using those funds to support CER through K and T mechanisms may be an excellent way of advancing the nation’s CER goals.

Additional comments from workgroup members to consider in future deliberations:

• • An enumeration exercise might be useful at some point. How many individuals at each institution are engaged in each type of CER activity?
• • Surveys about efforts to hire and difficulty hiring in some particular areas may be useful.
• • Data on the disciplinary training of people working in specific areas might be helpful. For example, how many investigators in cost‐effectiveness research have been trained as economists versus other disciplines. This may help to inform recruitment and training initiatives.
• • It may also make sense to think more explicitly about government and private sector demands for various CER skills.
• • Further data on earnings for various CER disciplines might help to grow and stabilize the field.
• • The lack of high response to many of the needs in Table 1 suggests that many CTSAs may not appreciate the importance of these competencies. For example, for only 19% to perceive a need for systematic reviews and 9% for new data analytic methods, suggest a potential lack of appreciation across CTSAs for certain CER competencies.
• • Related to some of the specialized CER skills noted above, which are currently not widely disseminated across a large proportion of CTSA institutions, Jonas and Croft recently suggested mechanisms to augment training at a national level. ⁹ These include “expanding efforts to disseminate methods work (e.g., a program offering certification in systematic review methods, perhaps delivered via short courses at national meetings), expanding training programs within individual EPCs/institutions, developing a collaborative training curriculum across Evidence‐based Practice Centers, and increasing the existing infrastructure for government‐funded training grants, such as AHRQ training grants and career development awards…‘On‐the‐job training’ for newcomers to CER may be another strategy.”
• • One of the scarce resources in CTSAs is mentoring support. If new fellows are funded through K or T awards, there needs to be sufficient faculty to provide consultation, to facilitate education and training, and to foster career development. Currently, this is often provided by faculty as a free service and it threatens to overwhelm the programs as they grow. Thus, there need to be mid‐career K awards for both clinical and nonclinical faculty as well as nongrant support for such work when it is not supported by other means.