How do we measure the implementation of the World Health Assembly resolution on rheumatic fever and rheumatic heart disease in African countries? Rationale and design of an evidence-based scorecard

Summary

Background

This article aims to explain the rationale and design for developing an evidence-based scorecard to monitor country-level implementation of the 71st World Health Assembly resolution on rheumatic fever and rheumatic heart disease (RHD) in Africa.

Rationale

A scorecard provides a simple and reliable tool for tracking progress over time and establishing accountability mechanisms.

Methods

Development of the scorecard will incorporate engaging RHD stakeholders identified and categorised at a country level. We will conduct individual interviews to understand the barriers and facilitators to implementing the resolution. The Delphi technique will facilitate structured group discussions to develop appropriate indicators. Indicators will be linked to the resolution’s goals to create strategic lines of action, to inform the scorecard. The scorecard will be quantitatively validated in real-life settings.

Discussion

We deem that the rigor of the development process of this study will produce an evidence-based scorecard that is fit for purpose across Africa

In May 2018, the 71st World Health Assembly (WHA) adopted a resolution on rheumatic fever (RF) and rheumatic heart disease (RHD),^1,2 emphasising the need to treat RHD as a health priority and to establish specific actions to address the needs of people with RHD. Ratified by 26 World Health Organisation (WHO) member states and six non-governmental organisations,^1-3 the resolution calls on governments to improve access to primary healthcare services for RF and RHD; strengthen data collection and knowledge of RF and RHD prevalence in endemic countries; ensure affordable and reliable access to technologies and medicine for RF and RHD; strengthen national and international co-operation and networking for RF and RHD; and tackle the root determinants of RF and RHD.² Even though the WHA resolution on RF and RHD is not a binding legal document, member states are expected to report back on the progress made in implementing these recommendations,² with the first reportback expected to take place in 2021 at the 74th WHA in Geneva.²

Monitoring will be the backbone of achieving the recommendations of the WHA resolution. Given that no specific mechanism for monitoring has been proposed to date by the WHO, we propose the development of a scorecard that may be used by African countries and beyond, to evaluate the implementation of the WHA resolution on RF and RHD. This work will not duplicate any efforts by the WHO and other organisations working on this resolution. Instead, it is an addition to efforts in the African region and globally to assist in implementing the resolution.

Here, we present the rationale, together with the justification for the approach to developing this scorecard. The study is being conducted by both the University of Cape Town and the South African Medical Research Council. Ethical clearance has been obtained from the University of Cape Town, Faculty of Health Science Research Ethics Committee, and the study is noted by the South African Medical Research Council, Research Ethics Committee.

Why the need for a global resolution against RF and RHD?

Globally 33 million people live with RHD,⁴ a chronic inflammatory heart disease.⁵ RHD is a result of recurring episodes of RF, which result from an autoimmune response to infection with Group A Streptococcus (pharyngitis) (GAS).^5-7 RHD leads to nine million disability-adjusted life years lost and 275 thousand deaths annually.^4,8 In contrast to the neareradication of RHD in high-income countries since the 1980s,⁹ there is a high burden of RHD in low- and middle-income countries (LMICs), making it the leading cause of cardiac disease among children in LMICs, and the third most common cause of heart failure in Africa.¹⁰

Reasons for this high burden in LMICs, especially in Africa, include inadequate data on RHD disease burden, limited knowledge on cost-effectiveness of RF/RHD prevention, and poorly developed health information infrastructures,^9,11 against a background of low awareness of RHD, consistently low priority for addressing RHD burden, weak health systems and persistent poverty.^11-13 The absence of adequate data and tracking systems have meant that there are insufficient data and limited knowledge on the successes or failures of health systems approaches, and biomedical and socio-economic approaches to dealing with the burden of RHD in LMICs.

Rationale for monitoring

Monitoring is a systematic method that is used to continuously collect data such as inputs, activities, outputs and outcomes for a specific programme, intervention, policy and/or goal.^14,15 The formulation of a monitoring framework for RHD will require a clear understanding and description of the objectives of the goals, the actions necessary to fulfill the goals, and the data needed to track these goals. This clarity could guide policy makers and decision makers on policy and programme options.¹⁶

The formulation of a monitoring framework for RHD may contribute to improved decision making at a political level (helping decision makers realise the scale of the problem) and at an implementation level (helping managers better organise the delivery of interventions). A monitoring framework can also be used to co-ordinate and align national and global leaders around a common approach to country support and reporting requirements, thus reducing the duplication of efforts for the same intervention, policy, programme and/or goal.

Once the monitoring framework is designed and implemented, the continuous collection of data can be routinely used to help policy and decision makers assess what has been done, to what extent, what has been achieved, and what needs to be improved to achieve all outlined goals.^14,15 Hence, data collected through monitoring frameworks could provide local and global decision makers and funders with essential information that is required to demonstrate results, secure future funding, and enhance the evidence base for interventions.¹⁶

Furthermore, the continuous collection of data will also enhance transparency in the decision-making process, while creating a culture of accountability at all levels.¹⁶ In addition, the data can be used for comparing progress across multiple countries; cross-country assessment has the potential to foster healthy competition and promote evidence-based decision making through lesson sharing.¹⁶

There are different forms of monitoring frameworks, one of which is a scorecard. A scorecard is meant to be a visual display of the goals, objectives, inputs and outputs of activities. A scorecard allows for a simple and visually appealing manner to report on progress.^14,15

Rationale for the study design RHD stakeholders

To develop the scorecard, we will engage with RHD stakeholders who have expertise in RHD in the African context. A stakeholder will be defined as an individual who has a vested interest in any aspect of RHD prevention, treatment, management, policy and administration.^17,18 Engaging with RHD stakeholders across the African region holds the potential to have a scorecard that is appropriate for the African region.

RHD stakeholders often have a deep understanding of RHD and are grounded in an understanding of the context of their local resources, demography and political landscape, and this may be used to inform the development of the scorecard. Recently, RHD stakeholders within Africa have worked alongside international and global partners and colleagues to organise themselves and work around a common purpose. They have worked to raise the awareness of RF and RHD,^19-21 improve the quality and availability of RF and RHD clinical and surveillance data,^22-24 and advocate for funding and the establishment of RF and RHD national prevention policies.^25-27 Their work, lobbying and advocacy have paved the way for RHD to be recognised on the global stage and have the resolution taken up. Therefore, it is essential to engage with them in the development of the scorecard as they have important and varied sources of information and resources.

A flow chart of the full process is depicted in Fig. 1.

Fig. 1.

The flow chart describes the process of developing a validated scorecard for monitoring in country implementation of the WHA resolution on RF and RHD. We will demonstrate the identification and categorisation of RHD stakeholders at a country level. RHD stakeholders will be engaged with through in-depth interviews and a group discussion to develop the content of the scorecard. The scorecard will be validated in a real-life setting.

Identifying the stakeholders

We previously conducted a systematic review in two African countries, Tanzania and Uganda, including in our published protocol an objective to identify RHD stakeholders in these two countries.²⁸ In the original review, we were not able to identify any articles identifying RHD stakeholders in the two countries.²⁹ Therefore, to identify and categorise RHD stakeholders who need to be engaged with, we will be guided by the Schiller et al. framework.¹⁷

Schiller et al. developed a framework in 2013 to systematically identify stakeholders related to older adult mobility and the built environment.¹⁷ The process involves searching the literature to identify stakeholder categories, engaging with experts to obtain practice-based insight and then representing the identified stakeholders in a visual chart.¹⁷

We will search for published and unpublished literature, conference reports and other conference proceedings, opinion pieces and narrative reviews on any aspect of RHD in Tanzania and Uganda. From these sources we will extract details of the researchers, field workers and anyone who was involved in that study. We believe these sources of data will reveal researchers, fieldworkers, data collectors and others who were involved in the study. We will also search for RHD policy documents and RHD reports from non-governmental organisations and ministries of health.

From these sources of information, we believe that we will be able to extract information on stakeholders, such as policy and decision makers in the ministry of health, public health planners, programme managers, programme co-ordinators, patients and civil society. Furthermore, we will contact in-country RHD stakeholders to provide us with insight and access to data sources that may not be available or accessible online.

Stakeholders identified will be assembled into a master list of stakeholders tabulated per country. Stakeholders within institutions will be characterised on the level of department or division within an institution. These stakeholders will then be thematically organised, using a pile-sorting approach, into five groups characterised as: (1) the general public and civil society, (2) educational sector, (3) research, training and capacity building, (4) healthcare service delivery, and (5) policy and administration. These broad thematic categories will be further sub-divided into smaller stakeholder groups. The stakeholders will also be categorised into local and international groups. The findings will then be displayed in spoke-and-wheel diagrams, flow charts or narrative form, depending on the nature of the data.

We will then apply these identified categories to ensure that all RHD categories are sampled across Africa. This will be supplemented by a snowballing technique, in which we ask each of the stakeholders we engage with to suggest additional RHD stakeholders who have worked or work in the African region.³⁰

The scorecard

We plan to engage with previously identified RHD stakeholders throughout Africa to gather input from a variety of perspectives and opinions on the key elements that need to be included in such a scorecard. African countries share certain similarities when it comes to RHD, such as the low priority of RHD and low funding.^31-33 They also differ greatly in many respects, including the epidemiology of RHD, resources dedicated to RHD and the ability of the general health system to deal with RHD. The successful adoption and acceptance of the scorecard will largely depend on how its framework fits within the environment of the countries.

To date, the eastern Mediterranean region of the WHO (EMRO) has developed a framework to guide the eastern Mediterranean countries on key steps that can be followed as a way of implementing the WHA resolution for RF and RHD.³⁴ This framework was developed through a technical consultative meeting that was attended by global and regional experts, such as representatives from government and non-governmental organisations from the EMRO, and representatives from the World Heart Federation and WHO regional office for the eastern Mediterranean.³⁵

From the EMRO framework we learnt that, even though countries in the EMRO are diverse politically, economically and demographically, using insight gained from experts from the region, they were able to develop a framework that is applicable to countries in the region despite their differences. This holds promise for using insight from RHD stakeholders across the African continent to similarly develop a scorecard that can be applied in multiple diverse settings, that is universally exchangeable in information, but remains locally relevant.

Developing the scorecard

To gather data for the scorecard, we will engage with stakeholders using two different processes: (1) through individual interviews to understand the barriers and facilitators to implementing the resolution in their context, and (2) through a structured group discussion to develop a list of appropriate indicators for the scorecard.

Qualitative inquiry

We will conduct in-depth individual interviews with the aim of understanding, describing and explaining the contexts across the African region in which the WHA resolution on RF and RHD will be implemented. The interviews will be conducted face to face, while some may also be conducted using analogue (telephone) or digital (Skype, WhatsApp or GoToMeeting) calling systems. In-depth interviews will provide rich data that highlight individual perceptions and understanding of their context and the WHA resolution.³⁰

Specifically, we are interested in exploring stakeholders’ perceptions of the contextual opportunities for, and barriers to, achieving the recommendations outlined in the WHA resolution for RF and RHD. This is important in ensuring that the scorecard adjusts for national contexts and implementation capacities.³⁰ By identifying potential influential factors in the implementation of the WHA resolution, it may contribute to the development of a more applicable and effective scorecard.

Structured group discussion

We will conduct a group discussion where RHD stakeholders will deliberate on which indicators should be included in a scorecard that can be used by African countries to monitor progress in the implementation of the WHA resolution of RF and RHD at a country level. To initiate the group discussion, we will present the context barriers and facilitators that were gathered from the in-depth interviews. This will ensure that stakeholders take into cognisance the context when they suggest and discuss indicators for the scorecard.

Additionally, we will conduct a ranking exercise to prioritise the suggested indicators by asking the stakeholders to rate the suggested indicators on the level of importance. This will allow us to develop a final list of indicators. Based on the diversity of participants, we hope that the discussion and consensus achieved will reflect stakeholders’ priorities towards eradicating RHD in the African region.

The final list of indicators will provide building blocks that are basic components for evaluating the implementation of the WHA resolution on RF and RHD. To develop a scorecard, these indicators will be linked to the WHA resolution on RF and RHD objectives and all recommendations/goals, thus creating a scorecard with strategic lines of action for implementing each of the recommendations of the WHA resolution on RF and RHD.

The structured group discussion will be conducted through the Delphi technique, which is a structured process that uses a panel of stakeholders to investigate an issue that requires group agreement.^36-38 The Delphi technique involves a feedback process that allows and encourages participants to continuously re-assess their judgements about the information provided, while providing the researcher more scope to follow up for clarification.^37,38 The Delphi technique allows for anonymity, thus allowing experts to freely express their opinions without undue social pressure for what is deemed as the acceptable opinion.^36,39 The whole process will happen online with no physical meetings or workshops, thus providing RHD stakeholders an equal chance to participate in this discussion without limitations due to time, travel requirements and financial resources for travelling.

Piloting the scorecard

The developed scorecard will be quantitatively validated to ensure that it adequately and appropriately monitors the implementation of the WHA resolution on RF and RHD within a real-life setting. We will assess both the internal and external validity of the scorecard in the context of two African countries, Botswana and Namibia. We choose these countries because they represent LIMCs that have endemic RHD, local RHD champions and reasonable data mechanisms.

To collect data, we will develop two questionnaires based on the scorecard developed. The first questionnaire will test the scorecard validity based on participants’ experiences with RHD in their local context. The second questionnaire will test the scorecard validity based on routinely collected RHD data in public health facilities (two clinics and two hospitals) in the chosen countries. The public health facilities will be chosen to allow for a mix between rural and urban facilities.

The following will be assessed:

• Indicators are not ambiguous. They do not make it difficult to measure what is intended and make it impossible to obtain consistent results.

• Necessary information is available in routine databases.

The results of this study will enable the researcher to amend and finalise the scorecard.

The scorecard will be piloted in only two countries for feasibility and practical reasons. The diversity of health systems and context that is found in African countries cannot be comprehensively captured through data collected in only these countries. Therefore, these data are not intended to be generalisable and any inferences made from these findings will be treated with caution. However, we believe that we will be able to gather rich data on a small-scale exploration of the scorecard’s ability to measure the progress of the resolution’s implementation. Furthermore, we will also gain insight into routinely collected RHD data in these chosen countries.

Discussion

This article presents the rationale and design for developing a scorecard for measuring country-level implementation of the WHA resolution on RF and RHD in the African region. We anticipate this validated scorecard will provide a simple and reliable tool for tracking progress over time and establishing accountability mechanisms when it comes to RHD.

Our approach of a mixed-methods research study is expected to yield in-depth data by bringing to the surface what cannot be derived from a single method. We deem that the end-process of this research will result in a scorecard that is collectively owned and fit for purpose across the African region because of the rigor of the developmental process. In addition, the research process will enable us to contribute to the scientific knowledge of RHD through the production of articles and other published outputs.

First, we will draw upon a wide range of data sources and insights from in-country experts, to comprehensively and accurately identify and describe stakeholders involved in RHD in two East African countries. This work will generate RHD stakeholder categories that should be engaged with when designing and implementing RF and RHD policies such as the WHA resolution on RF and RHD.

Second, we will engage with RHD stakeholders across Africa through individual in-depth interviews to explore the context of where the scorecard will be used. Understanding the situation within countries has the potential of developing a scorecard that is relevant and applicable. Additionally, this can potentially provide important data that could be used as a baseline for monitoring progress in the implementation of the WHA resolution on RF and RHD in different African countries.

Third, we will conduct the Delphi technique to facilitate a group discussion and ranking exercise among the stakeholders to develop indicators for the scorecard. Based on the diversity of participants, we believe that the consensus reached will provide indicators that are essential and are a priority across the African region.

Fourth, we will pilot the scorecard to assess its ability to monitor the implementation of the resolution in a real-life setting through data collected in two African countries that represent LIMCs. This will allow us to produce a tested scorecard.

Limitations

Our process has several limitations. First, we need a broad range of data sources to identify and categorise stakeholders. This will require us to manage a wide variety of literature from different disciplines. We anticipate that lower-quality studies will be important sources of information for public health planning, so we will have to relax the usual stringent criteria used, for example, systematic reviews of clinical trials. Second, the scope for our study requires the data to be collected from stakeholders working in different countries across Africa. However, we believe that our study methods that include virtual and in-person data collection will enable us to reach many of the relevant stakeholders.

Dissemination of findings

The final scorecard will be published on a relevant website that will be easily accessible to all RHD stakeholders. Plain language summaries in the form of policy briefs will be submitted to national health policy makers, decision makers, programme managers and funders to ensure that they are aware of the scorecard and its validity to monitor the implementation of the resolution.

In addition, at the end of the study, a report of the main study findings will be shared with all stakeholders who took part in the study. The findings will also be communicated through academic publications and conferences. We will also use the known GAS, RF and/or RHD networks to disseminate the information, which will enhance the uptake of the research. This dissemination strategy will allow the research to reach the appropriate audiences, which include national and global government authorities, researchers, funders, as well as the RHD community and other stakeholders at which the research is targeted.