EVALUATION OF A TRANSDISCIPLINARY CANCER RESEARCH TRAINING PROGRAM FOR UNDER-REPRESENTED MINORITY STUDENTS

Abstract

Aim/Purpose

This paper will describe the initial development of systems to evaluate research education activities of a U.S. academic Partnership to train minority students as cancer researchers and provide preliminary data from monitoring Partnership activities during the first six months.

Background

There is increased focus on multidisciplinary/transdisciplinary research training programs. However, few training programs have included detailed evaluations to assess their outcomes and effectiveness.

Methodology

The Temple University/Fox Chase Cancer Center and Hunter College Regional Comprehensive Cancer Health Disparity Partnership (TUFCCC/HC Cancer Partnership, or the Partnership) is a recently-initiated center focused on training individuals from under-represented minorities (URMs) as cancer researchers. Evaluation of the training activities involves detailed specification of goals for each of the Partnership’s Cores; objectives for addressing each goal; measures and indicators to determine progress towards each objective; and data sources to provide information for the measures/indicators.

Contributuion

This paper will provide important information for evaluation of training programs focused on students from URM populations and development of trans-disciplinary research education programs.

Findings

Goals, objectives, measures, and data sources for evaluation of the Partnership’s Research Education Core (REC) were developed jointly by personnel from the REC and the Planning Evaluation Core (PEC) in an iterative process. These measures capture the training activities led by the REC (e.g., number of seminars and workshops), scientific output by trainees (e.g., papers and grants), and ability of the program to meet trainees’ goals and expectations.

Recommendations for Practitioners and Researches

Evaluation plans for transdisciplinary training programs need to be developed prior to program initiation. Evaluation measures should be jointly specified by training and evaluation personnel, then reviewed and revised in an iterative process.

Impact on Society

This program is intended to enhance diversity among cancer researchers and increase studies to address disparities in cancer care.

Future Research

The PEC will oversee the evaluation of Partnership training activities over the five year period and assist Partnership leaders in ensuring successful outcomes.

INTRODUCTION

In the United States (U.S.), racial/ethnic minority subgroups bear a disproportionate share of the cancer burden. To achieve health equity, it is essential to understand the factors that underlie and drive cancer health disparities. A well-trained, diverse scientific workforce is a key component in addressing health disparities; however, investigators from diverse backgrounds are underrepresented across a number of fields, including biomedical research.

The Temple University/Fox Chase Cancer Center (TUFCCC) and Hunter College (HC) Regional Comprehensive Cancer Health Disparity Partnership (TUFCCC/HC Cancer Partnership, or the Partnership) is a recently-initiated center focused on training individuals from under-represented populations including racial/ethnic minorities and first generation college students in cancer research. The overall goal of the Partnership is to establish an integrated collaborative research infrastructure between TUFCCC and HC that develops and supports cancer research, education and outreach to eliminate cancer health disparities in underserved populations. The TUFCCC/HC Partnership involves a stewardship model that broadens the scope of cancer training to create new pathways to careers; prepares students and junior faculty to participate in multidisciplinary endeavors; engages scientists more fully in science and society: and recruits from traditionally underrepresented populations.

This paper provides background information on the Partnership; describes the initial development of systems to evaluate Partnership research education activities to train cancer researchers; and discusses preliminary data from monitoring Partnership training activities during the first six months.

METHODOLOGY

PARTNERSHIP MEMBERS

The Partnership includes:

Hunter College (HC).

Hunter College is a comprehensive teaching and research institution serving approximately 23,000 students, of which 70% are ethnic/racial minorities, 28% are firstgeneration college students, and 42% are recipients of federal need-based aid.

Temple University/Fox Chase Cancer Center (TUFCCC).

Temple University (TU) is a comprehensive public research university with more than 39,500 students enrolled in over 400 academic degree programs. Fox Chase Cancer Center (FCCC), which is affiliated with the Temple University Health System, is a National Cancer Institute-designated Comprehensive Cancer Center and a member of the National Comprehensive Cancer Network (NCCN).

PARTNERSHIP STRUCTURE

The Partnership is comprised of five cores:

Administrative Core.

The goals of the Administrative Core are to build on an effective leadership and governance infrastructure to provide administrative, scientific and programmatic oversight and monitoring, including timely coordination and communication, logistical support and budgetary management to ensure the successful achievement in high-quality standards of excellence in the proposed aims and objectives for all cores, pilot/full projects and shared resources.

Biostatistics and Bioinformatics Shared Resource Core (BBC).

The goal of the BBC is to facilitate and enhance research in the Partnership by providing expertise in experimental design, statistical analysis, bioinformatics, epidemiology, genomics, data management and statistical computing.

Community Outreach Core.

The goals of Community Outreach Core are to provide a community-based participatory research framework for the Partnership and develop and implement effective community outreach activities and intervention initiatives to reduce cancer disparities among underserved African, Asian and Hispanic American populations in the Eastern region of U.S.

Planning and Evaluation Core.

The goals of the Planning and Evaluation Core are to evaluate the short-term and long-term process and outcomes for the overall Partnership and each Core; to assess funded and new research projects; and, to provide recommendations to Partnership leadership to ensure the accomplishments of the overall goal of the Partnership.

Research Education Core.

The goal of the Research Education Core is to support educational activities that complement and enhance the training of a diverse workforce to meet the nation’s biomedical, behavioral and clinical research needs to address cancer health disparities.

RESEARCH EDUCATION CORE SPECIFIC AIMS

The Research Education Core (REC) addresses the following specific aims:

1. Develop and implement a comprehensive cancer research education program that provides research enrichment opportunities for undergraduate and graduate students from under-represented populations.

   • The Research Education Core will offer: (1) an 8-week intensive Summer Cancer Research Institute (SCRI) for undergraduate and graduate students in biomedical, clinical, or population-based cancer research; (2) ongoing skills-based training through mentored research projects; and (3) a rich curriculum of cancer seminars, workshops, journal clubs, and grand rounds throughout the academic year.
2. Provide mentorship and research support for URM cancer fellows (postdocs, junior faculty, or early-stage investigators) to enhance career development and facilitate their transition to independent investigators.

   • The Core will offer an established environment to support the career development of selected cancer fellows, who will receive close mentorship and support for two years, through the implementation of mentored research projects supported by pilot project funds.
3. Facilitate academic-community research opportunities to support the conduct of culturally appropriate cancer research in partnership with underserved minority communities and populations.

   • The Core will increase access to educational and research opportunities that utilize multidisciplinary approaches and engage underserved minority communities in addressing cancer health disparity research questions through collaborative academic-community partnerships.

PARTNERSHIP TRAINING ACTIVITIES

Summer Cancer Research Institute (SCRI).

The SCRI is an 8-week research intensive program that enables students to participate in a cancer curriculum and gain hands-on training in cancer research on the TUFCCC campus. Students will be matched with mentors for their research training. Trainees will attend seminars and will participate in mentored cancer research projects and skill-building workshops. At the conclusion of the program, students will be required to prepare a project abstract and oral presentation to their peers and to faculty advisors at the SCRI Symposium. During the SCRI, trainees can take advantage of other seminars and unique events that may be temporally relevant and occurring campus-wide.

Enriched training activities.

During the academic year, trainees will select specific enriched training activities in accordance with their individualized training plan and appropriate for their current needs and future career goals. These activities include:

• Cancer Seminars. TUFCCC sponsors several cancer seminar series including a campus-wide noon cancer seminar throughout the academic year, a monthly Distinguished Lecture Series, and a postdoctoral seminar series.

• Journal clubs and working groups. There are a number of topic-specific journal clubs that meet on a regular basis. Examples include journal clubs in Immunology, Cancer Biology, and Genetics. Similarly, a number of working groups meet regularly to address common themes including an Inflammation Working Group and a Health Disparities Working Group. The Health Disparities Working Group meets quarterly to address issues of health disparities in the local communities. Meetings include presentations by faculty and community partners on issues related to study design and community participation.

• Professional/skill development workshops. Trainees can participate in various workshops offered throughout the year. Prior workshop topics have included: presentation skills; grant-writing workshops; English as a Second Language course; staff management skills; and networking. Because communicating effectively to different audiences is askill that all trainees should develop, the Core will host workshops to enhance trainees’ communication skills and confidence in engaging with diverse audiences.

• Scientific meetings. Trainees will be encouraged to present their research at scientific meetings. The Core will support trainees’ travel costs and meeting expenses. National meetings also offer training workshops and professional networking opportunities.

Mentored research projects.

Trainees will participate in mentored research projects, which offer key experiences in laboratory training, conducting experiments, analyzing data, and reporting out-comes. This will provide trainees with skills in conducting literature reviews, practicing safe protocols in the lab, and using standard laboratory equipment or procedures. In combination with the training activities described above, implementation of these research projects will enable trainees to gain hands-on experience in developing a project from the initial idea, obtaining the appropriate regulatory (human subjects or animal use) approvals, data collection, and analysis and interpretation. The pilot data obtained from this project can be used to support applications for external funding. Trainees will also have multiple opportunities to present their findings to various audiences.

Both graduate students and postdoctoral fellows will have opportunities to participate in workshops and training activities on public speaking, professional networking, and effective mentorship. A major strength of the program is leveraging our strong network of secondary peer mentors — postdoctoral fellows and research associates — who can serve as peer mentors with shared experiences. These peer mentors provide not only informal mentoring but also social and peer support to help students develop strategies for addressing challenges.

FACULTY MENTORS

Core faculty provide mentorship and guidance to trainees on mentored research project proposals, career development plans, and grant applications. To ensure that trainees have multiple sources of support, a “team mentoring” approach is used, in which each trainee will have at least two faculty mentors. In addition, Core faculty facilitate community engagement, where appropriate, to increase the translational relevance of cancer research projects to underrepresented communities.

A Mentoring Advisory Committee (MAC) has also been convened. The overall function and responsibilities of the MAC are to: a) identify and assist with pairing mentees with appropriate mentors; b) provide oversight and guidance on mentorship activities; c) provide programmatic assistance to Core trainees and mentors, as needed; and d) facilitate career development and networking opportunities across institutions.

EVALUATION DATA COLLECTION

A key element for collection of data to evaluate the Partnership research training activities is to remove barriers (i.e., decrease burden) for submitting needed evaluation measures. This increases both the quality and the completeness of collected information (Holden & Zimmerman, 2008; Patton, 2009). To minimize the evaluation data collection burden, the Planning and Evaluation Core (PEC) uses a concurrent, real-time framework. Rather than delayed gathering of evaluation measures at later time points, with the potential accompanying recall bias, this permits collection of needed evaluation information with enhanced efficiency and quality. In addition, information on satisfaction with/utility of Partnership services, workshops, seminars, and other supports and events is collected via web-based anonymous survey and sent directly to the PEC. This encourages researchers, clinicians, trainees, and community members to provide honest and critical feedback regarding Partnership training activities and facilitate rapid quality improvement to address concerns.

FINDINGS

The Planning and Evaluation Core (PEC) collaborated with the REC to evaluate career development, mentorship, and other research education activities. The aims of the PEC include:

1. Establishing a planning and evaluation structural system to monitor progress, improve activities, and assist in achieving the overall goals of the Partnership to address cancer health disparities;

2. Establishing a systematic process to solicit and review new research project proposals to increase the competitive research capacity among TUFCCC/HC partners and promote research initiatives in cancer care for underserved minority populations, and

3. Conduct ongoing process, outcome, and impact evaluation of Partnership Cores and research projects to ensure proposed short-term and long-term outcomes are met.

The PEC oversees the evaluation of Partnership training activities and assists Partnership leaders in ensuring successful outcomes. The framework for the evaluation of Partnership research training activities involves three components:

1. Identification of Goals that describe the main research training activities of the REC;

2. Specification of Objectives to determine whether each Goal is successfully accomplished;

3. Listing Measures and Indicators used to assess progress towards completion of each Objective;

To guide and coordinate this planning and evaluation process, it was important to identify measures that appropriately indicate the status of processes and achievement of associated outcomes; collect data in an-ongoing process; analyze and interpret collected information; identify barriers, challenges, and unmet needs; and provide feedback to improve performance, increase quality, and provide greater effectiveness and efficiency. This is an iterative process, involving initial identification and collection of evaluation measures, followed by ongoing analysis and review of these measures; discussion of the relevance, completeness, and value (i.e., the time required or other burdens involved in collecting a measure relative to its importance to the evaluation) of measures; and refinement of the evaluation measures and data collection processes. The initial goals, objectives, and measures/indicators for the REC evaluation are presented in Table 1.

Table 1.

Goals, objectives, and measures/indicators for the REC Evaluation

GOALS	OBJECTIVES	MEASURES/INDICATORS
Develop and implement a comprehensive cancer research education program	• Establish admissions process for cancer research education program • Provides research opportunities for undergraduate and graduate students	Summer Cancer Research Institute [SCRI]: • Number of trainee applications and trainees selected for 8-week SCRI • Demographic characteristics of applicants and selected trainees • Seminars, workshops, journal clubs, and other training sessions offered in SCRI • Surveys examining achievement of educational goals and satisfaction among trainees • Trainee publications, presentations • Career milestones Enriched Training Activities: • List of cancer seminars • List of journal clubs and working groups • List of workshops and skill development training events offered Mentored Research Projects: • Number of mentors participating • Number of trainees • Demographic characteristics of trainees • Satisfaction of trainees with group events and individual-level mentoring programs • Scientific meetings/conferences attended by trainees • Trainee publications, presentations • Career milestones
Provide mentorship and research to enhance career development and facilitate transition of URM cancer fellows to independent investigators.	• Support cancer fellows at the postdoctoral, junior faculty, or early-stage investigator level	• Number of mentors available/participating • Demographic characteristics of selected fellows & research interests • Number of seminars, workshops, journal clubs, and other training/mentoring sessions offered • Surveys examining achievement of educational goals and satisfaction among trainees • Fellow publications, presentations • Scientific meetings/conferences attended • Grant proposals submitted, grants funded • Career milestones
Facilitate CBPR opportunities to support the conduct of culturally appropriate cancer health disparities research with underserved minority communities and populations.	• Provide educational offerings/events to support academic-community research collaborations	• Number of REC training sessions & workshops • Attendance by researchers and community members at training sessions & workshops • Number of academic-community collaborative projects • Publications and presentations • Scientific meetings/conferences attended • # of grant applications submitted involving academic-community partnerships; # of grants funded • Researcher/community partner satisfaction survey

Using this approach, the PEC evaluates both processes and short- and long-term outcomes. In addition, the PEC evaluates trainee-reported utility of career mentoring activities on development of competencies in team-based science and cancer communication skills. This includes use of a Self-Assessment of Interdisciplinary Research Competencies instrument (Gebbie et al., 2008). Responses to this self-assessment instrument are collected at initiation of the training program; self-reported deficiencies are used to individualize career development and mentoring programs for each trainee. This instrument is re-administered each year to assess changes in self-assessed “team science” competencies.

Results of the 6 month evaluation of the Partnership research education activities, based on the measures/indicators indicated in Table 1, will be presented at the conference.

CONCLUSIONS

This paper describes the structures and processes for evaluation of research training activities in the TUFCCC/HC Regional Comprehensive Cancer Health Disparity Partnership, a newly-initiated center funded by the U.S. National Institutes of Health (NIH) that focuses on training individuals from under-represented minorities (URMs) as cancer researchers. The Partnership involves a unique collaboration between two organizations with a shared vision of advancing cancer equity among underserved and underrepresented minority populations through training and research. This program is intended to enhance diversity among cancer researchers and increase studies to address disparities in cancer care. A strength of the Partnership is a dedicated Planning and Evaluation Core (PEC) to monitor progress towards achieving research education goals and objectives; identify challenges in realizing these outcomes; and provide recommendations to successfully address these challenges.

Over the next five years, the Partnership will educate students from under-represented populations in cancer research. The evaluation of these research education activities will help ensure that students, mentors, and community members benefit from their participation with the Partnership and that the Partnership’s aims are successfully completed.

Biographies

BIOGRAPHIES

Michael T. Halpern, MD, PhD, MPH, is Associate Professor and Graduate Program Director in the Dept. of Health Services Administration and Policy, Temple University College of Public Health. Dr. Halpern’s research focuses on health services, health policy, and program evaluation. This includes assessments of treatment patterns, quality of care, access to care, disparities, costs, and cost-effectiveness as well as evaluations of training programs and healthcare innovations. Dr. Halpern’s work is focused on social determinants of health, including patient, provider, and healthcare system-level factors that affect receipt of medical care and subsequent patient outcomes, particularly for individuals with cancer and other complex chronic diseases. Dr. Halpern is also a public health historian with a focus on changing role of public health during the early 20^th century. He has more than 160 peer-reviewed publications; serves on the American Society of Clinical Oncology’s Government Relations Committee and Education Committee; is co-chair of the Policy Subcommittee of the Pennsylvania Cancer Coalition; and is Section Editor for Disparities Research for the journal Cancer.

Sarah-Jane (SJ) Dodd, MSEd., MSW, PhD, is an Associate Professor, at the Silberman School of Social Work, Hunter College, CUNY, and the CUNY Graduate Center, and is also Director of the Silberman Center for Sexuality and Gender (SCSG). Dr. Dodd is a social work scholar with particular expertise in program evaluation and applied research. She has conducted program evaluations funded by the Department of Health and Human Services Office of Minority Health, NIH, SAMHSA, and the Children’s Bureau. Dr. Dodd has spent two decades addressing health disparities in the area of HIV/AIDS. She is the co-author (with Irwin Epstein) of Practice-Based Research in Social Work: A Guide for Reluctant Researchers (2012).

Dr. Carolyn Fang is Professor and Co-Leader of the Cancer Prevention and Control Program at Fox Chase Cancer Center in Philadelphia, PA. Dr. Fang received her PhD in psychology from the University of California, Los Angeles (UCLA) and completed her postdoctoral training in cancer prevention and control at Fox Chase Cancer Center. As a behavioral scientist with expertise in cancer control and health disparities, the primary focus of Dr. Fang’s research is investigating how psychosocial and behavioral factors may influence cancer risk and outcomes, particularly among racial/ethnic minority and underserved populations. She has led several collaborative endeavors to address health disparities in underserved Asian Americans and is currently the PI of several NIH-funded multidisciplinary grants that examine biobehavioral pathways contributing to health disparities and cancer risk. Dr. Fang serves on the editorial board of several journals, including Health Psychology and Translational Behavioral Medicine. Dr. Fang is an active member of the American Psychological Association (APA), the Association for Psychological Science (APS), the Society of Behavioral Medicine (SBM), and the American Society of Preventive Oncology (ASPO); and she recently served on the Executive Committee of ASPO and as Chair of the Lifestyle Behaviors Special Interest Group (SIG) in ASPO. In light of Dr. Fang’s commitment to leadership and service in her field, she was selected to join the American Psychological Association’s Leadership Institute for Women in Psychology in 2011. She has received various awards, including the American Cancer Society (ACS) Cancer Control Award in 2013 for her achievements in cancer control and health disparities.

Yin Tan, MD, MPH, is Senior Program Director and a senior researcher at Center for Asian Health (CAH), LKSOM, Temple University. Dr. Tan has an over thirty years’ experience in public health research. She is experienced in developing and implementing evidence-based cancer outreach programs and engaging community partners to culturally adapt and tailor these programs through a wide array of outreach strategies including community education intervention, media campaigns, patient navigation and community capacity building. She is also experienced in program evaluation.

Lin Zhu, PhD, is the postdoctoral associate, data manager and analyst at Center for Asian Health, Temple University. She has a broad background and research experience in social behavioral sciences, including medical sociology, cultural psychiatry, and epidemiology. Within the theoretical frameworks of health behavior model, theory of planned behavior, and biopsychosocial determinants of health, her work examines how social, cultural, environmental/contextual factors interplay to influence health behaviors and outcomes of individuals. Her substantive research primarily focuses on depression risk and symptomatology, cancer screening behaviors, and healthcare service utilization, especially with respect to acculturation models and population- or community-level factors. She has strong experience in research conceptualization, measurement operationalization, hypothesis formation, mix-methods study design, and multilevel evaluation.

Dr. Olorunseun Ogunwobi obtained a medical degree from the University of Ibadan, Nigeria, a master’s degree in biomedicine from the University of Hull, UK, a master’s degree in clinical and translational science from the University of Florida, Gainesville, USA, and a PhD in molecular medicine from the University of East Anglia, Norwich, UK. He is Director of the Hunter College Center for Cancer Health Disparities Research, Associate Professor of Biological Sciences at Hunter College of The City University of New York, and a member of faculty in the Biology and Biochemistry PhD programs at The Graduate Center of The City University of New York. Dr. Ogunwobi is a Contact Principal Investigator of the Synergistic Partnership for Enhancing Equity in Cancer Health (SPEECH) funded by a U54 grant from the National Cancer Institute in the USA. An author of numerous peer-reviewed research articles, Dr. Ogunwobi has also been issued two United States patents for biotechnology inventions with potential clinical applications in cancer.

Grace X. Ma, PhD is Associate Dean for Health Disparities, Founding Director of Center for Asian Health, Laura H. Carnell Professor and Professor in Clinical Sciences at Lewis Katz School of Medicine, Temple University. Dr. Ma is a nationally recognized behavioral health scientist and leader in health disparities research. Her community-based participatory research (CBPR) and patient-centered outcomes research (PCOR) focus on early detection, patient navigation, cancer prevention and control (Hepatitis liver cancer, cervical, breast, lung and colorectal cancers), chronic diseases (Hypertension and Diabetes), smoking cessation, and access/quality of healthcare in underserved Asian Pacific Americans and other diverse health disparity populations. Dr. Ma is the founding Director of Temple University’s Center for Asian Health, one of the first centers in the U.S. dedicated to reducing cancer and health disparities among underserved Asian Pacific Americans. Dr. Ma in partnership with community leaders co-founded the first Asian Community Health Coalition and Cancer Disparity Network in the U.S. Eastern Region, including Pennsylvania, New Jersey, New York City, Delaware and DC/Maryland. Dr. Ma has led large-scale randomized intervention trials, many using her innovative model for reducing health disparities and addressing health inequities in underserved populations. Dr. Ma has authored and co-authored 5 books, over 150 peer-reviewed publications and delivered over 560 professional presentations at regional, national and international conferences. The impacts of her accomplishments are reflected in academic teaching, mentorship, research, as well as in minority population health care policies and practices. Dr. Ma has trained and mentored over 158 faculty, postdoctoral, doctoral and master minority fellows to create a pipeline of diverse researchers and health practitioners to conduct health disparity research in ethnic populations.