The Importance of Evaluating Health Disparities Research

Abstract

Health disparity populations are socially disadvantaged, and the multiple levels of discrimination they often experience mean that their characteristics and attributes differ from those of the mainstream. Programs and policies targeted at reducing health disparities or improving minority health must consider these differences.

Despite the importance of evaluating health disparities research to produce high-quality data that can guide decision-making, it is not yet a customary practice. Although health disparities evaluations incorporate the same scientific methods as all evaluations, they have unique components such as population characteristics, sociocultural context, and the lack of health disparity common indicators and metrics that must be considered in every phase of the research.

This article describes evaluation strategies grouped into 3 components: formative (needs assessments and process), design and methodology (multilevel designs used in real-world settings), and summative (outcomes, impacts, and cost). Each section will describe the standards for each component, discuss the unique health disparity aspects, and provide strategies from the National Institute on Minority Health and Health Disparities Metrics and Measures Visioning Workshop (April 2016) to advance the evaluation of health disparities research.

Disparities in health among different populations account for substantial preventable morbidity and mortality, both nationally and internationally. Research in this area mostly has focused on identifying differences in health and on improving our knowledge of factors that contribute to disparities, with the hope of discovering interventions that will reduce them. Public Law 106-525(d) mandates that health disparities populations include minority groups, as defined by the Office of Management and Budget, as well as rural location populations, those of low socioeconomic status, and sexual or gender minorities.¹ Increasingly, research funders are emphasizing that the relevance of minority health and health disparities programs and public policies must be demonstrated to promote good population health and to justify program investments.^2,3 However, health research is expensive, and public expenditures supporting health research and health care are receiving increasingly close scrutiny. This heightened level of attention and debate requires that research identify not only health benefits but also social and economic impacts, which requires methodologies based on good evaluation techniques.

During April 2016, the National Institute on Minority Health and Health Disparities (NIMHD) convened a workshop on Methods and Measurement Science for Health Disparities to help inform strategic planning for the Institute. The evaluation of health disparities research was an important topic of the visioning workshop. Invited experts agreed that few evaluations were conducted as part of health disparities research, and this needed to be rectified. A conclusion from this workshop was that incorporating evaluation in health disparities research is needed to amass a rich evidence-based portfolio of interventions that can inform policymaking activities, as well as to justify program investments and substantiate their utility.

Health disparity populations and health disparities research present unique challenges to traditional evaluation because of the population characteristics, sociocultural context, and nascence of the field. Differences between health disparity populations and the “mainstream population” often result from social disadvantage, experience of discrimination through “isms” (e.g., sexism or racism), unique cultural practices, enclaves of living, and difficulty with established societal structures and bureaucracies. In addition, differences often depend on the degree of acculturation of the health disparities population into mainstream society.⁴ Therefore, evaluations must consider related population characteristics and sociocultural environments, but these considerations challenge the use of traditional or mainstream interpretations of standard evaluation techniques because of the implications of being a socially disadvantaged population.

A paramount consideration in health disparities research evaluations is to do no harm to the populations involved. Mainstream biases can lead to misinterpretation of results and questions about the quality and efficiency of a process, product, or outcome.⁵ Such effects can cause harm to the study population. For example, in a condom use study, Hispanic adolescents were declared less likely to use condoms during sexual activity when an evaluation of the project concluded that the adolescents did not use condoms because they were not having sex.⁶ The harm of promiscuity was not acknowledged by the evaluation. Population and sociocultural context must be at the forefront of every stage of the evaluation to ensure that appropriate results are given to decision-makers.

The field of health disparities research is maturing into a discipline, especially with the emergence of new definitions and better understanding of health determinants and relevant outcomes.⁶ Using findings from multiple disciplines, investigators are showing how health determinants contribute to health disparities at multiple domains of influence, with effects ranging from the individual level to the systemic. Examining the relationship of these factors has become the focus of health disparity etiology and intervention research. However, appropriate methodologies for evaluation in applied or real-world settings have not been commonly implemented and must be adopted.

Rossi et al. have defined program evaluation as the systematic application of scientific methods to assess the need, implementation, design, outcomes, and costs of a program.⁷ Key steps of an effective evaluation originate from their evaluation hierarchy, a well-known conceptual tool that identifies 5 sequential stages of a project. The 5 levels are the evaluation of project

• 1.need,
• 2.implementation and process,
• 3.design or theory,
• 4.outcomes and impacts, and
• 5.costs and efficiency.

These evaluation levels assess distinct phases of the program to provide appropriate information to decision-makers to improve, sustain, adapt, or end the project. In this article, we organize these 5 levels into 3 components: (1) formative, (2) design and methodology, and (3) summative. The following sections describe these evaluation components, the unique aspects that must be considered in health disparity research evaluations, and recommendations from the NIMHD Methods and Measurement Science visioning workshop.

FORMATIVE EVALUATIONS—GETTING IT RIGHT

Formative evaluation collects information to (1) determine need; (2) define purpose and questions to be answered; (3) establish process, develop the data collection plan, identify the information sources, and specify the instruments; and (4) form a feedback approach for program improvement.⁸ The information can be qualitative, quantitative, or both, and the methods of analysis can be descriptive or comparative.⁹ These types of evaluations are the first step in assessing health disparity research to help determine whether the research is appropriate for the population and sociocultural environment.¹⁰ A needs assessment determines the program’s requirements relative to the population and environment and determines potential gaps to be addressed. This step must ensure that the evaluation of health disparities research includes population characteristics and unique sociocultural constructs that often are not known to external researchers or are different from mainstream settings.¹¹ For example, a community health needs assessment is vital to identifying the health concerns of communities, to learn about the factors that influence their health, and the assets, resources, and challenges that affect those factors. Once these are identified, meaningful research can be conducted and evaluated to address health disparities locally and respond to longstanding historical health inequities at the health-system level.⁶

Needs assessments provide a fact-based way to determine gap areas and avoid assumptions of requirements based on mainstream standards. They also help identify and capitalize on the strengths of the health disparity population. Strength-based assessments enable communities to be seen in light of their abilities, talents, competences, possibilities, visions, values, and hopes.¹² They assume communities have the capacity to do their best, grow, and change.¹³ Both needs- and strengths-based assessments of health disparity populations should be mindful of multilevel social context and physical environments, such as system policies, laws, geographic location, existence of food deserts or environmental toxins, lack of resources or green space, and extent of crime. These external factors often are not considered in program evaluations but can have a substantial effect on the involvement of participants and in the success of health disparity interventions.¹⁴ Associated impacts often are subtle and missed, which may result in the health disparity population being blamed for inadequate implementation, poor outcomes, or both.

An implementation or process evaluation can be used to examine the fidelity of the implementation as it was planned to ensure the program is being conducted correctly to achieve the goals.¹⁵ This information documents the tools, instruments, processes, and outputs of the ongoing program and explores the relationship among design, implementation, and outcomes. It is applied at a time when changes can be made to ensure that the program is moving in the right direction to achieve its goals and enables stakeholders to see how a program outcome or impact was achieved.¹⁶

Population characteristics and sociocultural context must be considered in every implementation decision. Instruments should be assessed for language, literacy, and cultural appropriateness for the population(s) evaluated. For example, the Latino conceptualization of depression advocates that current instruments may not be adequate for Latinos because feelings, such as “blue,” are not interpretable into Spanish, and the Latino concept of depression differs from that expressed by Whites.¹⁷

To determine the best approach in assessing need and implementation, unique aspects relevant to the health disparity populations under study must be considered. These populations may be discriminated against, be underrepresented in research, lack sufficient resources, require time to identify and engage stakeholders, present unique psychosociocultural values, experience external assumptions of need when need is not perceived internally, project fear of the misuse of their personal information, and not have the knowledge or tools to combat system discriminations.^18,19 These experiences and perceptions differ from those experienced and expressed by mainstream populations, influence behaviors and attitudes, and frequently take considerable time to manage in real-world applied community research.²⁰ These factors must weigh into every aspect of the process evaluation so that decision-makers receive useful information and harm is not generated by misinterpretation through using a mainstream framework. For example, in a study on raising healthy children, researchers found that validated family-based behavioral interventions are an underused approach that has been found to significantly prevent excess weight gain and obesity in children and adolescents in minority populations, including Latinos, African Americans, and American Indians. Researchers acknowledged the specific needs of these communities in preparation of the study protocol before it was tested in the community, and the intervention was very successful.²¹

Researchers outside of a community may not know or understand the nuances of the culture, life experiences, system discrimination, and protective coping skills embodied by the population. These researchers may never penetrate the community defenses to engage the population in research or to conduct evaluations. Without gatekeeper and stakeholder engagement, important population and sociocultural elements of an evaluation may be underappreciated or missed, and important challenges may not be addressed as well as harmful misinterpretations generated.²² For instance, researchers assumed that African American women were not depressed and did not experience pain because they immediately returned to work after breast cancer surgery. Evaluation of the project showed that an African American woman recovering from a mastectomy may go back to work and take care of her children immediately after surgery because she has no sick leave or child care, not because she is not depressed or is not in pain.²³

It is suggested that researchers partner with community gatekeepers and stakeholders to interact with the community and to ensure that the research and evaluation is designed and implemented appropriately for the population. Thus, when working in disadvantaged communities where health disparities are more common, interacting with a variety of groups within the community from the onset is important. Involving local gatekeepers to identify population characteristics, the problem, the goals of the evaluation, and what resources are needed to accomplish the goals are the first steps in developing promising implementation strategies that actively engage community members as participants and yield relevant information for decision-makers.²⁴ These gatekeepers are key in providing feedback to the community and eliciting a sense of value for participating in research. Although gatekeepers often are not researchers, most can understand and participate in designing the methodologies that will best secure participants and assess the research findings. These partnerships are key in conducting and evaluating research in health disparity communities.²⁵

Formative evaluations foster relevant planning and implementation, as well as set the foundation for establishing the appropriate study design. The box on page S37 presents strategies for formative evaluations to facilitate success.

Strategies for Formative Evaluations.

Evaluate research during the development, implementation, and process phases to determine the best approach, modifications, or progress that are inclusive and reflective of the unique aspects of health disparity populations in real-world settings.

• •Conduct a needs assessment to ensure the research and the hypotheses tested address health disparity topics relevant to the needs and context of the health disparity population.
• •Foster strength-based assessments to draw upon the assets of the population to reduce health disparities.
• •Evaluate processes to ensure that the appropriate practices, protocols, and tools are being implemented to facilitate scientifically valid results relevant to the health disparity population (e.g., instruments, policies, timing, training, data collection, reliability of the practices, and adherence to the design).
• •Ensure a feedback mechanism to improve the project, identify components that were effective, and ensure that the design and outcomes are meaningful to the health disparity population.

DESIGN AND METHODOLOGY EVALUATIONS—DOING IT RIGHT

Health disparities research includes both etiological research and interventional research. The former seeks to understand the causes of health disparities, while the latter seeks to reduce them. Both need to be informed by sound methods, including evaluation. A separate article in this issue by Jeffries et al. (p. S28–S33) addresses methodological approaches to understanding the etiology of health disparities. Noting that potential causes are often correlated, occur at multiple levels, and may involve feedback loops, Jeffries et al. focus on design and analytic methods for causal inference from observational studies, discuss complex systems and simulation methods for modeling dynamic relations, and describe qualitative and mixed methods in addition to quantitative methods. Qualitative methods foster in-depth understanding of the culture of minority and vulnerable populations, support needs and resources, identify intervention preferences, and assess satisfaction with intervention strategies and impacts. Although labor-intensive and focused on individual voices and perspectives, qualitative methods can be combined with quantitative data to provide greater insight than either quantitative or qualitative methods alone. In a research program addressing low-income and elderly people in Canada, the use of mixed methods enhanced assessment of the mediating impacts of social support on the health of vulnerable populations and enabled the design and testing of support interventions.²⁶

Interventions often operate at multiple levels and in a community or other real-world setting that is dynamic and heterogeneous with regard to race, gender, education, poverty, risk behaviors, and physical environments. That makes it difficult to use randomized controlled trials for their evaluation, even though randomized controlled trials are widely considered to provide the strongest evidence for causal inference. Randomized controlled trials are best used when the researcher has considerable control over all aspects of the study, including the criteria for participation and recruitment of participants, the selection and timing of measurements, the delivery of the intervention, and the analysis of the data. Such control provides good internal validity, maximizing the opportunity for an unbiased result.^27–29 However, investigators may not be able to achieve the randomized controlled trial level of control for evaluations of interventions designed to reduce health disparities and instead may have a strong interest in achieving a high level of external validity.

Fortunately, a number of alternative methods are available and appropriate for the evaluation of multilevel and community-based intervention programs that are designed to reduce health disparities. The suggested designs that follow may account for complications, such as the challenges of minority recruitment, lack of sufficient research funds, interactions of multilevel health determinants, and the extended duration of community-based projects in real-world settings, as well as provide the needed rigor.

Group- or cluster-randomized trials (GRTs)^30,31 can be just as rigorous as randomized controlled trials but are designed specifically for interventions that operate at a group level, change the physical or social environment, or cannot be delivered to individuals without substantial risk of contamination. For example, neighborhoods or whole communities may be randomized to study arms, allowing a multilevel intervention to be delivered to the neighborhoods or specific sites within communities that are randomized to the intervention program.^32,33 Group- or cluster-randomized trials are the strongest design option if randomization is possible in conjunction with a no-treatment, delayed-treatment, or alternative-treatment control arm.

Stepped-wedge designs^34,35 also are well suited to test the effectiveness of multilevel or community-based intervention programs. In this design, the groups to be randomized are measured, and 1 or more groups are selected at random to receive the intervention. After an interval that is long enough to assess the intervention effect, measurements are repeated, and 1 or more groups are selected at random from those remaining to receive the intervention. This process continues until all groups have received the intervention and a final round of measurements is completed. The intervention effect is calculated by using the control and treatment observations taken in each group.

One of the primary advantages of this design is that all groups receive the intervention before the end of the study, a feature that will be important to many individuals and communities considering participation in an intervention to reduce health disparities, and this may make it easier to persuade both individuals and community groups to participate. Another advantage is that the design will usually be more efficient than the standard GRT, often requiring fewer groups. The disadvantages are that more measurement occasions will be required, perhaps increasing cost, and the study will usually take longer to complete than the standard parallel GRT, creating more opportunity for unexpected problems and delaying the final evaluation of the utility of the intervention.

The regression discontinuity design (RDD) is an attractive alternative when randomization is not possible.^28,29,36,37 In the RDD, participants are allocated to study arms on the basis of a quantitative score, with all participants on one side of a predetermined cutpoint receiving the intervention and all participants on the other side of that cutpoint serving as controls. Accurate modeling of the relationship between the assignment variable and the outcome in the primary analysis provides strong evidence for causal inference.³⁸ The major advantages of the RDD are that randomization is not required, and the intervention can be given to the participants with the greatest need. Such advantages would likely be attractive to participants considering a study to evaluate an intervention to reduce health disparities. The disadvantage is that twice as many participants are required for the same power available in a randomized controlled trial. Even so, the RDD has great potential in studies in which political or cultural reasons make randomization difficult.

The group-level RDD is patterned after the individual-level RDD described previously.^32,39 Groups rather than individual participants are allocated to study arms. This variation on the RDD design may be a useful approach to evaluate community-level health disparity interventions in which the intervention operates at a group level, changes the physical or social environment, or cannot be delivered to individuals without substantial risk of contamination. Analysis of RDDs can be accomplished with standard methods based on the general or generalized linear model.^36,37 Model-based analysis of GRTs, stepped-wedge designs, and GRDs will require application of the general or generalized linear mixed model. Two-stage models, 2-stage randomization tests, and methods based on generalized estimating equations can also be used, if implemented properly.^{30,31,34,35,39}

If none of these designs can be used, multiple baseline designs,^40,41 quasi-experimental designs,^28,29 and time-series designs^28,29 may be helpful. However, these methods will not provide the same level of rigor as GRTs, stepped-wedge designs, RDDs, and group-level RDDs. The box on page S38 provides a summary of some design strategies to consider for evaluations of health disparity research.

Strategies for Design and Methodology to Advance Evaluations.

Use design and analytic methods that reflect the complexities of multilevel, multidisciplinary health disparities research.

• •The next generation of health disparities projects should promote multilevel and multidisciplinary research across the life course in real-world settings.
• •Etiologic research should assess exposures and outcomes at multiple levels and reflect the unique characteristics and attributes of health disparities populations.
• •Intervention research should address identified etiologic factors at multiple levels and be evaluated by using design and analytic methods appropriate for multilevel interventions.
• •Whether the health disparities research is etiological or interventional, evaluation typically proceeds with both qualitative and quantitative data to understand individual experiences and health outcomes.

SUMMATIVE EVALUATIONS—RIGHT GOALS, RIGHT PRICE

Summative evaluation examines the impact of a program or intervention on the target group. Because it is used to assess whether the results of the program or intervention being evaluated met the stated goals, it is often considered the most important part of an evaluation. Summative evaluation is particularly important for health disparities because the 2 key questions in the research field today are (1) have interventions focused on the right factors that improved the health of the target group, and (2) have programs actually reduced the health disparity between the target and reference groups? Summative evaluation documents the degree of achievement of the project’s goals and outcomes. It enables researchers to quantify changes in resources used, which can help determine the overall impact of the program or intervention, and it promotes the assessment of comparative analyses with different projects to facilitate evidence-based decision-making.

Evaluation in health disparities should be designed to improve transparency, accountability, and sustainability of health equity. A summative evaluation can demonstrate accountability for a variety of stakeholders, including the community and funding bodies. Accountability generally spans 3 dimensions: (1) how efficiently health care resources are being used (fiscal accountability), (2) whether interventions are being delivered with fidelity and yielding the expected result (performance accountability), and (3) whether interventions exhibit government-brokered equity, transparency, and responsiveness (political or democratic accountability).⁴² Understanding performance for each of these accountability dimensions requires evaluation to ensure that investments, interventions, and policies are actually yielding their expected results—and are not imposing harm. The methods of summative evaluation are typically associated with quantitative assessments but can include qualitative methods as well. Processes include analyses of the outcomes, products, or data from the project, as well as information from reports. It can also provide additional information or perspectives from focus groups, including project participants and members of the community the program was intended to affect.

Substantial reduction in health disparities among underserved and minority populations will occur only if interventions and programs are sustainable and effective. Although cost analyses are important in summative evaluations, understanding effectiveness regardless of cost is equally important. For example, when a study summarized the outcomes and cost-effectiveness analysis of a cardiovascular disease risk-reduction program delivered to patients in federally qualified health centers by nurse practitioners and community health workers versus enhanced usual care, the authors did report costs per unit of improvement in low-density lipoprotein cholesterol, systolic and diastolic blood pressure, and glycated hemoglobin.⁴³ Although the authors concluded that their intervention could effectively reduce cardiovascular health disparities over a 12-month period, the study design was unable to show an actual reduction in health disparity between the targeted group and a healthier reference group. As a consequence, they were unable to answer 1 of the 2 key questions in health disparities research today—Did the program reduce the health disparity between the target and reference (healthier) groups? In this case, a summative evaluation could have provided contextual meaning to the absolute and relative outcomes, which are often used in assessing health disparities research but not always understood by lay users. It also could have validated that the conclusions accurately represented the findings and that the population characteristic and sociocultural environment were construed appropriately.

Economic evaluations can be conducted to determine whether a program is cost-effective relative to comparable programs or is cost-beneficial, which is helpful for choosing among programs to be funded and to foster sustainability. Although an evaluation should include an economic analysis that compares the costs and consequences of alternative interventions,⁴⁴ this is uncommon in health disparities research. An important aspect of an economic evaluation is specifying the perspective, which can range from individuals to health systems; a governmental program, such as Medicare; or society overall. The appropriate approach depends on what is being compared.

A cost-effectiveness analysis compares the likely health-related effects and costs of alternatives to achieve a specific health-related outcome. The resulting incremental cost-effectiveness ratio (net cost:net effects) may, for example, relate to incremental cost per life year saved or incremental cost per case averted. Including a strategy to determine the cost-effectiveness of minority health and health disparities interventions in community settings is critical to include in the design and long-term sustainability of funded projects. Because the outcome under study is life years saved or survival, cost-effectiveness analysis does not account for improvement in the quality of life of beneficiaries, which is so often associated with health equity interventions. Hence, cost-utility analysis is another type of economic evaluation that expands cost-effectiveness analysis to include multiple health-related effects, such as quality and length of life. Cost-utility analysis typically uses the preferences of individual patients for specific health-related outcomes and aggregates them to a societal level to develop an overall measure, such as a quality-adjusted life year.

Key considerations when one is assessing outcomes and impact include the potential scalability of the results to real-world settings where the implementation strategy can be expanded broadly across populations and the sustainability of health-promoting interventions. Evaluations must move health disparities projects to this point to reduce disparities and to save costs. Policymakers and other key stakeholders are often interested in knowing whether a particular program reduces cost, also known as “the return on investment.” In this case, a cost–benefit analysis that expresses both costs and outcomes in monetary terms of programs addressing health disparities is warranted. For instance, Delaware increased colorectal screening rates among African Americans and, unlike most health disparities projects, continued to assess the screening rates over time to determine whether the colorectal cancer disparity was reduced.⁴⁵ Overall, the intervention cost about $1 million per year and saved an estimated $8.5 million annually in treatment costs. Equally important, not only did health outcomes improve for both groups, but the health disparity between groups was also reduced. A basic evaluation investigating impact like this example rarely receives attention in health disparities research, although it is much needed.

Achieving “the right goals at that right price” is only part of the journey. Once projects, especially interventions, are properly evaluated, results must be communicated. Reporting summative evaluation findings encompassing assessments of goals and cost analyses is critical to expanding the health disparities literature. Such information can support systematic reviews and meta-analyses, encourage development of interventions that can affect health equity, facilitate evidence-based decision-making with regard to scalability and sustainability of programs, and reduce repetition. Another important activity is to provide feedback to the community that was studied to foster collaborative, caring relationships and to diminish the perceptions of being exploited and concerns of funding consequences. In addition, a centralized location is needed to promote access to health disparities research findings. An example is HDPulse, which is hosted by NIMHD (https://hdpulse.nimhd.nih.gov). Such a strategy could advance health disparities research in a meaningful way by cataloguing exemplary evaluation approaches and other science-driven activities to encourage research in health disparities populations to evolve at a quicker, yet deliberate pace. Moreover, it can become a repository not only for cost-effectiveness outcomes but also for quantifying improvements in health and in health disparities reduction—both of which answer the most compelling questions in health disparities research today. The box on this page provides strategies for summative evaluation of minority health and health disparities research to help achieve the right goals.

Strategies for Summative Evaluations.

Evaluate research outcomes and efforts to determine the overall impact and relative costs of health improvement and reduction of health disparities for the targeted population.

• •Research often focuses on discovering significance. To progress from efforts directed toward accurate identification and surveillance to delivering effective interventions, health disparities research must incorporate summative evaluation methodologies to determine whether appropriate short-term outcomes and long-term impacts were achieved.
• •Interventions should be evaluated to determine whether they were delivered with fidelity and yielded the expected result (performance accountability) relevant to the population and sociocultural context.
• •Economic analysis should always be a part of the evaluation. It requires collection of cost data throughout the course of an intervention to determine the sustainability and feasibility of long-term use in relation to the reduction of health disparity.
• •Communicating the results is critical. Publishing evaluations about health disparity research will support systematic reviews and meta-analyses, which can lead to a better understanding of the impact of activities designed to reduce health disparities. Providing feedback to participant communities fosters collaborative relationships for future research.

CONCLUSIONS

The science of health disparities is promoting research from a comprehensive perspective of the individual, the community, and systems over the life span. The complexities of this work challenge not only conventional population science and research but also evaluation at every phase. Health disparities research must be encouraged to take on these challenges by promoting and adopting new discoveries in methodologies and data science to move beyond describing and understanding health disparities toward identifying mechanisms that can effectively and efficiently reduce them. A substantial part of this effort will require not only the implementation of evaluation methodologies but also the publication of findings from the evaluation.

The use of evaluation techniques in health disparities research is key to identifying the “active ingredients” that make health disparity interventions effective. Evaluation can play a dual role by providing evidence on the outcomes of the intervention within different settings and by appraising the intervention process to help clarify how and why the intervention worked and its cost. Projects lose long-term impact when the cost is too high to sustain. Routine evaluation can not only strengthen the position of minority health and health disparities programs so that investments can be justified when budgets are constrained, but also facilitate progress toward reducing health disparities and improving health and well-being for all.