Surveying the knowledge landscape in sub-Saharan Africa: methodology

Abstract

Objective

To describe in detail the methods followed in each of the qualitative and quantitative surveys of national health information, research and knowledge systems and research institutions.

Design

Cross-sectional surveys.

Setting

National health information and research systems, and 847 health research institutions in 42 countries in the World Health Organization (WHO) African Region.

Participants

Key informants from health research institutions, ministries of health and statistical offices.

Main outcome measures

Stewardship, financing, ethics, human and material resources and output of health information and research systems.

Results

Key informants were used to collect data to assess national research systems in 44 countries in the Region. The same method was followed in assessing national information systems in 17 countries and knowledge systems for health in 44 countries. These assessments included a detailed review of the state of data sources in the Region and their effect on measuring progress on the health-related Millennium Development Goals. A concurrent survey employed a structured questionnaire (the WHO Health Research Systems Analysis Questionnaire) to assess the capacity of 847 health research institutions in 42 countries. Stewardship, financing, ethics, human and material resources and research output were assessed.

Conclusions

The logistics and resources involved in surveying the knowledge landscape in the 42 countries were substantial. However, the investment was worthwhile as the results of the surveys can be used to inform policy-making and decision-making, as well as to establish a regional database of national health information, research and knowledge systems.

Introduction

A health research system has been defined as the people, institutions and activities involved in the generation and dissemination of knowledge. The health research system is an integral part of the health system. It should produce research findings that are synthesised into high-quality information and knowledge, and inform the development and strengthening of the health system. A systematic assessment of existing health research and knowledge systems will inform the development of policies and strategies to strengthen national health research systems. Such analyses were conducted and covered issues of governance/stewardship, ethical oversight, financing, resource inputs, outputs, dissemination and impact of national health research findings.

A health information system (HIS) has been described as an integrated effort to collect, process, report and use health information and knowledge to influence policy-making, programme action and research. There have not been comprehensive reviews of the state of national HISs in the World Health Organization (WHO) African Region, particularly the state of their inputs (resources), processes (selection of indicators and data sources; data collection and management) and outputs (information products; information dissemination and use).

Knowledge directly affects human health, and an effective health system needs effective knowledge management. Knowledge management, in this context, includes how to apply known interventions when resources exist, how to allocate limited resources for these known interventions according to identified priorities, and how to secure additional resources. These issues are of direct relevance to the African Region, where the know-do gap is mostly a consequence of inadequate capacity to apply existing knowledge. This may include, among other things, insufficient trained human resources, networks and communities of practice, and infrastructure. Knowledge management provides a set of methodologies and tools to overcome precisely these challenges. There have been no studies that assess the role of knowledge in the health system.

Countries and their partners have recognised the importance of evidence in shaping policy-making and decision-making in the Region and have been trying to improve their health information and research systems for some time.¹ In August 2006, at its 56th session, the WHO Regional Committee for Africa (comprising ministers of health from all 46^a countries of the Region) requested the WHO Regional Office for Africa to map the knowledge landscape of the Region by collecting core information and indicators on the current status of health research, information and knowledge systems. This evidence would serve as a key input to the Algiers Ministerial Conference on Research for Health in 2008. The surveys were conducted in 2007–2008 (Table 1). This was the first time that such mapping had been done with such scope and depth in the Region. This paper describes the processes and methods that were employed in the surveys, including the limitations of the tools and methodology.

Table 1.

Major areas of enquiry, type of survey and target population included in the various surveys, 2009.

	Area of study	Type of survey	Target population
	Health research
1	Description of national health research systems	Key informant	WHO country-level focal persons for research
2	Stewardship or governance of research at research institutions	Self-administered questionnaires	Senior officers at health research institutions
3	Ethics practice at health research institutions	Self-administered questionnaires	Senior officers at health research institutions
4.	Expenditures on research by health research institutions	Self-administered questionnaires	Finance officers at health research institutions
5	Human resources at health research institutions	Self-administered questionnaires	Human resource officers at health research institutions
6	Institutional facilities available at research institutions	Self-administered questionnaires	Administrative officers at health research institutions
7	Research output from health research institutions	Self-administered questionnaires	Senior officers/researchers at health research institutions
	Health information
8	Describe national health information systems	Key informant	Focal persons for information at ministries of health and statistical offices
	Knowledge management
9	Describe the knowledge system in the health sector	Key informant	Mid-level managers at ministries of health

Methods

Surveying national health research systems

In September 2006, the Regional Office asked each WHO Country Office representative to designate a country focal person for health information, research and knowledge management. In December 2006, these focal persons from 46 countries of the Region were invited to a workshop in Brazzaville, Congo, for orientation on the Regional Office’s activities on health information, research and knowledge management, and for training on the main tools that were to be used for collection of data during the surveys.

The health research surveys employed two tools, focusing on the national and institutional levels. The national-level survey used a tool originally employed for a similar survey in the Region.² The institutional survey employed a tool developed by WHO as part of its Health Research System Analysis Initiative.^3,4 The survey on knowledge systems for health used two survey processes (all tools used for the collection of data are in online supplementary material):

• One in which key data regarding knowledge systems were collected from international data sources, the results of which are not included in this report

• A second in which a questionnaire-based survey was conducted to collect data about the use of knowledge in health policy and clinical services provision

The Health Research System Analysis Initiative tools included questions grouped into seven modules.⁵ Each module covered a specific topic:

1. Identification, introduction and background information covered basic background information, including mission, governing structure and leadership, and organisational structure.

2. Types and approaches to research focused on the institution’s involvement in research, including the types and topics of research conducted, the type of study methods used, institutional priorities for health research, research activities, scientific review, monitoring and evaluation, collaboration with other actors, and balanced research partnerships.

3. Financial resources for health research aimed to collect data on financial resources spent or received by the institution for health research topics.

4. Health research output, synthesis, dissemination and knowledge management sought to measure the outputs, synthesis, dissemination and knowledge management of health research.

5. Human resources, capacity and development for research covered human capacity and development, training for research, and staff movement.

6. Institutional facilities and field sites focused on institutional facilities and field sites.

7. Research ethics and ethical processes collected information on the institution’s policies and concrete approaches to reviewing ethics and ethical processes, in terms of new research activities and ongoing research activities.

The tool was initially pilot tested by WHO in 10 low- and middle-income countries, including three African countries (Cameroon, Senegal and the United Republic of Tanzania). The tools were translated into French and Portuguese before their use in the surveys. Questionnaires in English were sent to institutions in 19 countries, in French in 19 countries and in Portuguese in four countries.

The main criterion for considering an institution as a ‘health research institution’ was that it should be engaged in the conception or creation of new knowledge, products, processes, methods and systems related to any aspect of health, such as factors affecting health and ways of promoting and improving it. Institutions could be departments of medical schools, universities, teaching or non-teaching hospitals, independent research institutions, governmental agencies, pharmaceutical and other for-profit and not-for profit businesses, charities and non-governmental organisations.

The WHO initially created a database of research institutions in the Region from various sources in order to build a reasonably comprehensive inventory. Sources used included:

• An Internet search

• International directories such as Thomson Reuters Web of Knowledge, the Nature Yearbook and the World Higher Education Database

• The Institute for Scientific Information databases (Science Citation Index, Social Science Citation Index, Art and Humanities Citation Index)

• An international health research journals subscription database

• WHO sources such as the Health InterNetwork Access to Research Initiative (HINARI) Phase 1 and 2; WHO collaborating centres; WHO research activity database; WHO Medical Schools Directory; the Global Forum for Health Research, Forum 4 and 6 participants; the WHO master mailing list, expert advisory panel

• International funders, including the USA National Institutes of Health grants and awards to foreign countries data

• Existing data and directories of ministries of health, science and technology at the national level

The list of health research institutions corresponding to a particular country was then sent to the country focal team to verify and confirm whether the listed institutions still existed and were operational, and whether they had carried out research on health or health-related topics in the past five years. A regular exchange with the coordinating team at the Regional Office to rectify discrepancies and inconsistencies, and add or remove relevant or irrelevant institutions, eventually led to a final list of all institutions conducting health research in countries, totalling 1882 institutions in the 46 countries of the Region. Of these, a total of 847 institutions conducting health research in 42 countries were included in this report. The number of institutions in each country that responded to the questionnaire ranged from 2% in Rwanda to 100% in Guinea Bissau. Overall, 45% of the institutions listed on the sampling frame responded to the survey (Figure 1).

Figure 1.

Distribution by country of the 1882 institutions identified as a sampling frame for the survey (graph on the left) and percentage of institutions that responded to the survey (n = 847) in 42 sub-Saharan African countries, 2009.

It was decided to include all of the research institutions in the survey, thus this was more of a census than a survey. This decision was taken because it was envisaged that national and regional databases of health research institutions could be created that could later be used by national groups and the Regional Office to formulate national policies and strategies to strengthen national health research systems. Regular monitoring and evaluation of national health research systems would also be possible by establishing national and regional observatories based on such exhaustive data.

Angola and Sierra Leone did not participate in the survey. Four other sub-Saharan countries (Djibouti, Somalia, South Sudan and Sudan) were also not included. Algeria and South Africa (both of which are part of the WHO African Region) were excluded, as their health research institutions are far more numerous and stronger in many respects than other countries in the group and thus their inclusion would have distorted the overall results. Of the total of 847 institutions in 42 countries included in the analysis, most were relatively young institutions (aged under 30 years), government agencies or hospitals and had as their primary mandates academic, research or giving services. French was the primary language for communication of over half of the institutions (Table 2).

Table 2.

Characteristics of health research institutions in 42 sub-Saharan African countries, 2009.

Characteristics	Health research institutions
	No.a	%
Age of institution (years) (n = 694)
<30	426	61
30–59	200	29
≥60	68	10
Sector the institution belong to (n = 762)
Public	536	70
Private not-for-profit	132	17
Para-state	37	5
Private for-profit	26	3
Other	31	4
Type of institution (n = 847)
Government agencies	257	30
Hospitals	154	18
Medical schools	108	13
Independent research institutions	106	13
Other research institutions (non-governmental   organisations, charities)	105	12
Other universities	95	11
Other	22	3
Level at which institution functions (n = 751)
National	483	64
Local	140	19
Regional	60	8
International	55	7
Other	13	2
Primary functions of institution (n = 697)
Conduct research on health topics	374	54
Academic	373	54
Provide health services	338	48
Conduct research on non-health topics	122	18
Product development or distribution	74	11
Other	128	18
National official or working language (n = 847)
French	445	53
English	285	34
Other	117	14
Institution has mandate on
Research of all types	571	79 (n = 723)
Health research	563	77 (n = 731)

^aNumber of respondent health institutions, out of 847 surveyed.

The modules were designed to be self-completed and involved several individuals working together within an institution to gather and discuss the responses. The modules were sent to heads of institutions by email or fax or given to them personally by the consultant. Each module was designed to be filled out by the appropriate specialist in the institution. The head of the institution had the responsibility of designating the appropriate specialist to fill in the relevant module and noted their names on the last page of the first module. In most questions, the survey focused on explicit policies and actions within the institution, rather than the opinions of the respondent.

Once collected, data were entered at the country level and shared online using DataCol (the WHO data entry program), which was easily accessible to WHO Country Offices via the Internet. Each country focal point had full access to the data that they entered and could make modifications in case of errors. The coordinators checked the data on a daily basis for completeness and consistency and, in consultation with country focal points and national consultants, made the necessary modifications. After the completion of this process, the data were exported and processed in a Microsoft Excel format and were available for further analysis in IBM® SPSS® Statistics Version 19.

Surveying national HISs

The national HIS surveys were managed and carried out by the WHO African Regional Office and the Health Metrics Network. The assessment focused on a comprehensive approach to HIS described in the Health Metrics Network’s and WHO’s Framework and Standards for Country Health Information Systems.⁶ The Framework serves two broad purposes:

• At the country level, it aims to focus investment and technical assistance for the standardised development of HIS and serves as a baseline for HIS assessment and diagnosis, a roadmap for the development of HIS plans, and provides for ongoing monitoring and evaluation

• At the country and global levels, it enables access to and use of better health information

The HIS assessment questionnaire comprised about 200 items organised in components: resources, indicators, data sources, data management, information products dissemination and use (see online supplementary material). The assessment tool was in a Microsoft Excel format and key informants discussed and scored each item. The results of the assessment of each component were generated as tables and graphs. These results were discussed and shared with stakeholders.

The stakeholder group and its steering committee designated an existing agency, such as the HIS section or unit of the ministry of health, to carry out certain administrative tasks (e.g. communications and procurement) required to conduct an assessment. The assessments were carried out in the setting of a workshop or meetings of several groups, or with individual interviews of key informants, or a combination of all these for soliciting inputs from all key stakeholders.

The two important mapping steps in the HIS assessment were the mapping of all stakeholders, partners and donors that were willing to strengthen their HIS, and the mapping of all the components of the HIS in order to develop consensus on priority weaknesses. The list of stakeholders included producers, users and financers at different levels (subnational versus national) of health information and other social statistics.

Countries have included in the assessment:

• Representatives of bilateral donors and organisations of the United Nations system active in development and in monitoring progress towards the Millennium Development Goals such as United Nations Children’s Fund, United Nations Development Programme, United Nations Population Fund and WHO

• The World Bank

• Representatives of key non-governmental organisations and civil society

These stakeholders were aware that the HIS assessment would very soon be followed by a comprehensive strategic planning process to which they would also be asked to contribute.

Surveying national systems on knowledge in health

A number of data items relevant to systems on knowledge in health are not available from international sources. In some countries, the data are not available at all, while in others, much or all of the data characterising their knowledge systems can be found in their national health information or national health informatics systems, public health associations and other institutes and bodies. Regional and country focal points were aware of such locally published data. In some countries, all or most of the data were available from locally published sources. For items where there were no locally published data, we opted to ask key informants in each country to give us an estimate or an opinion.

The initial survey focused on analysing knowledge systems in national health system layers dealing with:

• Health policy formulation, implementation and monitoring

• The provision of clinical services (our definitions for these layers of the health system are given below)

The questionnaire (in online supplementary material) sought to collect the opinions and estimates of experts on those items for which we found no published sources. We tried to eliminate questions for which published data already exist. Some of the questions in the questionnaire were matters of fact, while others were matters of opinion. At this stage in our efforts to describe the knowledge systems, we collected both qualitative and quantitative data.

The questionnaires were developed by WHO, making use of the advice of in-house survey and questionnaire specialists. As the original intention was to use them for a global survey, the questionnaires were then field-tested through full-day consensus workshops. The first such workshop was held in Prague (Czech Republic), and then in Addis Ababa (Ethiopia), Sana’a (Yemen), Manila (the Philippines), Bangkok (Thailand) and New Delhi (India). After these workshops, the questionnaires were adjusted and finalised for wider distribution. The health information, research and knowledge management focal point in every country in the Region received the questionnaires and collected data, making use of key informants.

We considered the health policy layer of the national health system as including all the elements involved in policy development, implementation and monitoring in the health sector. In a given setting, an event (an identified problem) triggers the policy deliberation. The process, which is not linear and involves interactions among different stakeholders at all times, often ends with a policy decision (budget, regulation, institutional arrangement), which may or may not be implemented and subsequently evaluated for improvement.

A country’s health policy may be centrally originated and managed, or decentralised to subnational entities (regions, provinces and states) or to bodies operating at national or even local levels (professional associations, syndicates or unions). If policy-making is evidence-based, the evidence may include research results; feedback from and studies of experience and implementation, guidelines, norms and standards developed internationally, regionally or locally; or other sources.

The questions aimed to clarify the knowledge translation process relating to health policy in a given country, identifying the principal stakeholders, actors and flows. Specific questions addressed the:

• Responsibility and type of work of the person completing the questionnaire

• Bodies and groupings involved in health policy formulation

• Role of the ministry of health and other central policy-setting and implementing bodies

• Sources and types of evidence used

• Dissemination and use of knowledge about health policy

• Knowledge about monitoring and evaluation of health policy

The second questionnaire covered knowledge in the provision of clinical services at the local level. Direct patient care accounts for a significant portion of the health enterprise in countries. Clinical practice includes various preventive or care interventions (e.g. education, diagnosis, medical or surgical treatment, follow-up) for individuals in settings such as medical offices, hospitals or regular home care (note that in-and-out, point-of-service interventions or commodity delivery to populations were not covered here).

The questions aimed to clarify the knowledge translation process relating to clinical services provision in a given country, identifying the principal stakeholders, actors and flows. Specific questions addressed the:

• Responsibility and area of work of the person completing the questionnaire

• Structure of delivery of clinical and health services in the country

• Knowledge related to policy and procedures

• Education and networking

• Knowledge sources and targets

Conclusions

The logistics and resources involved in surveying the knowledge landscape in the countries of the African Region were substantial. However, the investment was worthwhile as the results of the surveys can be used for:

• Convening national consultations to develop or reinforce strategic frameworks and plans

• Holding subregional and regional consultations with stakeholders and the development of strategic frameworks

• Establishing a regional database of national health research systems and the African Health Observatory

In future surveys, more time and effort would be required to ensure that the surveys are completed by as many institutions and key individuals as possible.

Declarations

Competing interests

None declared

Funding

WHO Regional Office for Africa

Ethical approval

Not required because the survey did not touch on ethical issues requiring individual consent.

Guarantor

DK

Contributorship

DK wrote the paper; CK and PEM co-wrote the paper; PEM, IS and WK reviewed the paper and assisted with fieldwork; and PSLD reviewed the initial design of the study and provided support and overall leadership.

Acknowledgements

The authors acknowledge the WHO Country Office focal persons for information, research and knowledge management for their contribution in coordinating the surveys in countries. They also acknowledge their counterparts in the ministries of health. These surveys would not have been possible without the active participation of the research and academic institutions and non-governmental organisations who have given their time and effort to complete modules and questionnaires.

Provenance

Not commissioned; peer-reviewed by Roderick Viergever