Data for development in health: a case study and monitoring framework from Kazakhstan

Abstract

Healthcare reforms are often not coupled with a relevant and appropriate monitoring framework, leaving policymakers and the public without evidence about the implications of such reforms.

Kazakhstan has embarked on a large-scale reform of its healthcare system in order to achieve Universal Health Coverage. The health-related 2020 Strategic Development Goals reflect this political ambition. In a case-study approach and on the basis of published and unpublished evidence as well as personal involvement and experience (A) the indicators in the 2020 Strategic Development Goals were assessed and (B) a ‘data-mapping’ exercise was conducted, where the WHO health system framework was used to describe the data available at present in Kazakhstan and comment on the different indicators regarding their usefulness for monitoring the current health-related 2020 Strategic Development Goals in Kazakhstan.

It was concluded that the country’s current monitoring framework needs further development to track the progress and outcomes of policy implementation. The application of a modified WHO/World Bank/Global Fund health system monitoring framework was suggested to examine the implications of recent health sector reforms. Lessons drawn from the Kazakhstan experience on tailoring the suggested framework, collecting the data, and using the generated intelligence in policy development and decision-making can serve as a useful example for other middle-income countries, potentially enabling them to fast-track developments in the health sector.

Key questions.

What is already known about this topic?

• Healthcare reforms are always complex and thus require feedback about their impact.

• Healthcare policy initiatives are often not coupled with a relevant and applicable monitoring framework.

What are the new findings?

• The existing national monitoring concept was reviewed and found to have substantial shortcomings in the health policy aspects covered and the usefulness of indicators.

• A simplified framework was presented, coupling national needs and the comprehensive monitoring framework developed by the WHO, the World Bank and the Global Fund.

Recommendations for policy

• Initiate a dialogue about the appropriateness of the suggested framework with national counterparts, develop national capabilities and apply the simplified framework to feedback results.

Introduction

Reforming a healthcare system is always a complex undertaking, characterised by conflicting interests and limited evidence, and it is often not known whether well-intended initiatives will improve people's lives.¹

Since such complexity and uncertainty can be found in most development and reform projects, the United Nations has launched a large-scale ‘data revolution for sustainable development’ in order to provide the right information for designing, monitoring and evaluating effective policies.² The WHO, in collaboration with the World Bank and the Global Fund, published a comprehensive operational framework on the monitoring and evaluation of health systems strengthening.³ In 2010, the WHO presented ‘Monitoring the Building Blocks of Health Systems: A Handbook of Indicators and Their Measurement Strategies’.⁴ In 2014, the WHO and The World Bank published a framework for monitoring Universal Health Coverage (UHC) as part of a comprehensive framework for monitoring national health system performance at national and international levels with guiding principles and methodological considerations.⁵ The need for encompassing Civil Registration and Vital Statistics as a core component of sustainable development has also been emphasised.⁶ The importance of effective and comprehensive measurement of change in health systems is increasingly being recognised.

In this paper, the health reform experience in Kazakhstan is used as a case study to explore what data are needed for the meaningful monitoring of reforms. The findings contribute to the global debate on measurement and data production for development in the health sector.

The following section (Theoretical background and methods) provides a theoretical background both for monitoring the implications of policy changes and the case study approach taken. Context: healthcare in Kazakhstan section then sets the context, giving a brief overview of the Kazakh health system and recent reforms. Next, the findings of the monitoring framework as applicable to Kazakhstan and available data are presented (section Findings). The discussion (section Discussion) presents the development of a comprehensive yet manageable monitoring and assessment framework. Finally, in the Conclusion section, we conclude what could be learnt from the Kazakh experience and provide the outlook for further developments.

Theoretical background and methods

Monitoring the implications of policy changes—a framework

Given that health policy is frequently not evidence-driven,⁷ policymaking regarding the provision and financing of healthcare is often as much based on the perceptions of politicians, the influence of advisors and personal empirical evidence from healthcare systems elsewhere as it is on rigorous analysis and systematic identification of strengths and weaknesses of the status quo. Furthermore, the ‘measurement of success’ of health financing and delivery arrangements is still in its infancy.¹ Figure 1 depicts the most general approach to looking at a healthcare system, starting from ideas and taking outcomes as the final goal.

Figure 1.

Ideas, interests and institutions in healthcare. Source: Modified after.¹

Several approaches have been developed to establish a conceptual framework with a clear distinction between objectives and instruments, that is, between what the system does and what the system is trying to achieve.⁸ One of the most widely used frameworks is based on (A) the WHO's six building blocks of any healthcare system, a mix of inputs, processes and management issues⁴ and (B) the core functions of a health system: Improved health (both level and equity), responsiveness and financial protection (see figure 2).^{9 10} This framework is simple yet allows one to capture structures, processes and results.

Figure 2.

The WHO Health System Framework with building blocks and goals outcomes. Source: Adopted from.⁴

Formal progress towards UHC assessed by intermediate goals will not automatically mean better population health,¹¹ so measuring effects at the population level will be as important as measuring intermediate indicators like equitable financing and access.^{12 13} Resonating with its strategy of consolidating and unifying data collection and evaluation for health systems worldwide, the WHO in collaboration with the World Bank and the Global Fund in 2009 published a comprehensive operational framework on the monitoring and evaluation of health systems strengthening based on the International Health Partnership (IHP+) common evaluation framework (figure 3).^{3 5}

Figure 3.

The results framework for health system strengthening monitoring and evaluation. Source: Modified from.³

This concept was taken up again in 2010 with a handbook on suitable indicators and measurement strategies ‘for monitoring performance and evaluating progress in the scale-up for better health’.⁴

The recent UN initiative on the ‘data revolution’ has put forward the importance of data for sustainable development. ‘Data are the lifeblood of decision-making and the raw material for accountability. Without high-quality data providing the right information on the right things at the right time; designing, monitoring and evaluating effective policies becomes almost impossible’.²

Method: the case study approach

We use the case study approach¹⁴ primarily to describe the current context and need for policy monitoring in Kazakhstan. On the basis of our personal involvement and experience, we (A) assessed the indicators in relation to national policy in Kazakhstan and (B) conducted a data-mapping exercise, where we used the WHO health system framework to describe the data available in Kazakhstan and to comment on their usefulness in augmenting the health-related 2020 Strategic Development Goals in Kazakhstan.

‘The case study approach allows in-depth, multi-faceted explorations of complex issues in their real-life settings.’¹⁴ It has only recently become more widely used in health service research. Early examples of the case study approach in health policy have been more descriptive with little theory¹⁵ or have emphasised the personal involvement and experience of the authors.¹⁶ Later research has tried to provide a more nuanced theoretical grounding and combined methods from the social sciences with an epistemological reflection and established processes for ensuring rigour in case study and qualitative data collection and analysis.^{17 18} We used publicly available data together with non-classified information that has not been published. In addition, we conducted a literature search in PubMed, Google Scholar and general Internet search engines with the combination of key words ‘Kazakhstan’ AND ‘public health’ OR ‘health care’ OR ‘epidemiology’ AND ‘data’ OR ‘monitoring’.

The case study team based the assessment of the current health-related indicators in the national 2020 Strategic Development Goals on three categories. A traffic light system was used with red indicating that the target could only be assessed subjectively and would thus be prone to discussion and criticism; yellow indicated that some element of verifiable assessment was possible, whereas green indicated clear (and often easy) measurement. To the best of our knowledge, such an assessment has not been done elsewhere, but this categorisation is self-explanatory and can easily be replicated. There is a need for such assessment of usefulness and measurability because unspecific, unclear and simply unmeasurable indicators are counterproductive.

Context: healthcare in Kazakhstan

Kazakhstan is a middle-income country that aims to become a member of the Organization for Economic Cooperation and Development (OECD) by 2050. The national strategy ‘Kazakhstan 2050’ has set strategic goals for further development of the country in all areas of social and economic development.¹⁹ It is one of the least densely populated countries in the world with only six people per square kilometre on average. Kazakhstan's Gross Domestic Product (GDP) was United States Dollar (USD) 218 billion, or USD 12 600 per capita, in 2014.²⁰ The country's GDP is to a large degree dependent on mineral resources, and thus the economy is particularly prone to external shocks. However, the country has a low public debt of just 12% of GDP.²¹

Kazakhstan's current state of health and health care provision

Compared to countries in the OECD and even fellow post-Soviet states, key population health indicators in Kazakhstan are at the lower end (see table 1). However, the government of Kazakhstan has acknowledged this and has made progress in the field of healthcare provision an explicit target of state policy.²²

Table 1.

Population health indicators in selected former Soviet Union countries/Eastern European countries

Country	Life expectancy at birth (years) [2012]	Infant MR (per 1000 live-births) [2013]	Maternal MR (per 100 000 live-births) [2013]
Moldova	68.7	13.3	21
Kazakhstan	69.6	14.6	26
Kyrgyz Republic	70.0	21.6	75
Russian Federation	70.5	8.6	24
Ukraine	70.9	8.6	23
Georgia	73.9	11.7	41
Bulgaria	74.3	10.1	5
Romania	74.6	10.5	33
Macedonia, FYR	75.0	5.8	7
Hungary	75.1	5.2	14
Serbia	75.2	5.8	16
Poland	76.8	4.5	3
Croatia	76.9	3.8	13
Czech Republic	78.1	3.9	5
Slovenia	80.1	2.3	7
OECD average	80.4	4.1 (2011)	11 (2010)

The World Bank Database (2014)—rounded values, OECD (2015), infant mortality rates, maternal mortality rates, doi: 10.1787/83dea506-en; doi.org/10.1787/888932722981 (Accessed on 27 October 2015).

FYR, Former Yugoslav Republic; MR, mortality rate; OECD, Organization for Economic Cooperation and Development; TB, tuberculosis.

The country inherited a Semashko-style healthcare system, which has an integrated approach to providing and financing care with primary healthcare and the district physician at the centre, a strong referral system and an extensive hospital network. From 1970, the system was not able to respond to demand for better quality care and new medical technologies. A move towards specialisation in outpatient care led to a much-reduced role of the district physician and direct access to specialist care, but this resulted in unmet demand for quality care.²³ At the same time, despite substantial reductions in the past decade, hospitalisation rates remain well above OECD levels.²⁴ Very soon after independence from the Soviet Union in 1991, private medical practice was allowed in Kazakhstan and the country started an ill-fated attempt to introduce mandatory health insurance (1996–1998).²⁵ Since 2000, the country has initiated two comprehensive reform programmes: the National Programme for Health Care Reform and Development 2005–2010 and the State Health Care Development Programme for 2011–2015 (‘Salamatty Kazakhstan’) with the specific aims of reducing the size of the hospital sector and strengthening primary healthcare.²⁵

Funding health care

At present, health funds are collected by the central government through general taxation and then either used for paying providers directly or transferred to regional authorities to cover medical services for ‘socially important diseases’ (like tuberculosis). In addition, there are specific healthcare budgets at the local authority level.²⁵

National level budget expenditures amount to 62.9% and local budget expenditures amount to 37.1% of total government health expenditures (2013 National Health Accounts (NHA) by the Ministry of Healthcare and Social Development and Republican Centre for Health Development, latest NHA not yet published). When comparing Kazakhstan's expenditure on health with other Eastern European, post-Soviet countries or the OECD average, Kazakhstan stands out as a country that spends a relatively low share of its GDP on health (see table 2). It seems that the provision of universal high quality services with this level of funding might pose a serious challenge.

Table 2.

Total health expenditures in selected Former Soviet Union countries/Eastern European countries and OECD average (2012 Data)

Country	Total health expenditure (as % of GDP)
Kazakhstan	4.2
Romania	5.1
Russian Federation	6.3
Poland	6.7
Croatia	6.8
Kyrgyz Republic	7.1
Macedonia, FYR	7.1
Bulgaria	7.4
Ukraine	7.6
Czech Republic	7.7
Hungary	7.8
Slovenia	8.8
Georgia	9.2
Serbia	10.5
Moldova	11.7
OECD average	8.9

The World Bank Database (2014)—rounded values; OECD health expenditure database.

The total health expenditure is around 3.8% according to the latest National Health Account.

FYR, Former Yugoslav Republic; GDP, Gross Domestic Product; OECD, Organization for Economic Cooperation and Development.

The government of Kazakhstan is aware of the below-average healthcare spending. A substantial increase in the health budget as well as a comprehensive budget review has been identified as one of the key factors for improving financial protection and better service levels in the Kazakh health sector.²⁶

Healthcare reform

There is a strong political will to pursue health sector reform. With multiple levels of immediate (Salamatty Kazakhstan 2011–2015), medium-term (Vision 2020 and 2030) and long-term (Kazakhstan 2050 Strategy) policy plans,²² there is also a clear time frame with defined objectives for health sector reform. While reform efforts are aimed at all building blocks of the healthcare system (K Obermann, T Chanturidze, B Tokezhanov, et al. Reforming the health system in Kazakhstan: A review based on WHO's concept of ‘building blocks’. Manuscript submitted), one central theme is the introduction of shared responsibility for health.²⁷ In this context, shared responsibility for health refers to the idea that workers, employers and the government all participate equally in the financing of healthcare and the maintenance of public health and healthy lifestyles. A key reform element is the plan to introduce a mandatory contribution component, akin to a SHI, into the current tax-financed health system. The country is not seeking to introduce SHI with the objective of completely redesigning the financing of its healthcare system, but rather the augmentation of an existing healthcare system with elements of a SHI scheme.²⁸ If the reforms are to maintain their momentum and culminate in sustainable improvements, dedicated measurement of change is needed as well as timely interpretation of the results that allows these changes to be explained and suggests future action.

Findings

The government of Kazakhstan released a ‘National Health Strategy 2020’ approved by Presidential Decree No. 922 of 1 February 2010^{22 25} with the primary aim of improving the accessibility and quality of health services. This is to be done via specific healthcare investment plans, broader health financing and more efficient provider payment systems. Healthy lifestyles are supposed to be an integral part of overall health policy. The Kazakh government is determined to measure the impact of its policies and has detailed its ambitions in the ‘Program Realisation Indicators’ of the Salamatty Kazakhstan Programme.

The goals of the strategy are reasonable and could be expected to result in improvements in the health of the population. However, looking at the concrete targets defined in the strategy, it becomes clear that they are for the most part rather imprecise. Most aim only for an increase or decrease in a certain indicator or the implementation of a policy, without specifying the current baseline and the degree of improvement that is desired. For example, internal data show that one of the productivity indicators of the NHA, the ‘number of operations per 1 million Tenge’, had a tendency to fall between 2010 and 2013. This could either be interpreted as a drop in efficiency or, quite to the contrary, be connected to the improvement of hospital infrastructure, allowing for more complex and expensive operations. A detailed technical analysis is required to ensure that policymakers interpret these indicators appropriately.

The assessment of health-related indicators in the 2020 Strategic Development Goals is based on their usefulness and measurability as described above (red: target can only be assessed subjectively and will thus be prone to discussion and criticism; yellow: some element of verifiable assessment is possible; green: clear—and often easy—measurement) and is presented in table 3.

Table 3.

Kazakhstan's health-related 2020 Strategic Development Goals

OOP, out of pocket payments; SGBP, state guaranteed benefit package; ALOS, average length of stay.

Whereas population health is well covered overall, there is no mention of equity between different population groups. The possible financial burden when seeking healthcare is also not covered in the national health strategy. However, both issues are extensively discussed in the social health insurance (SHI) concept and the draft SHI Law, as submitted to the Parliament, specifically aims to address financial equity in the system.

In the data mapping exercise (see table 4), we describe the data available at present in Kazakhstan and, using the available evidence, comment on the different indicators regarding their usefulness to augment the current health-related 2020 Strategic Development Goals in Kazakhstan. Data are in principle available in Kazakhstan for most of the measurements recommended by the WHO. Data are currently lacking for the General and Specific Service Readiness Scores and for the Health Information System Performance Index. A Health Facility Assessment has been conducted in order to obtain data on the Readiness Scores, but as of October 2015 no data were available. These data would also provide up-to-date information on the availability of essential medicines, which is a key objective of the 2020 Strategic Development Goals, and would allow for regional differentiation of workforce and infrastructure data. The Health Information System Performance Index so far has not been used.

Table 4.

Monitoring the implications of health policy changes in Kazakhstan based on the WHO health system framework: a summary of available data and suggestions for the health-related 2020 Strategic Development Goals

Measurement aspect	Available data	Comments based on the available evidence
Governance
Willingness for reform (eg, availability of laws, policy documents)	Salamatty Kazakhstan (2011–2015); Kazakhstan Social Development Concept of 24 April 2014 (Resolution №396); Presidential address from 17 January 2014	There is no clear consensus on the appropriate values for these indicators in any given health system. Should not be included
Trust in government institutions (general/health)	Survey data needed
Public expenditure tracking (PETS) in place	No PETS study carried out so far	To be carried out in order to assess flow of funds
Per cent of budget with defined responsibility central/local level	Overall: 62.9% central level, 37.1% oblast level (2013 NHA by MHSD and RCHD)	No added policy value
% admin cost of total government health spending	Not available	Helpful indicator, administration cost need to be estimated
Corruption Perception Index (general/health-specific)	General: Transparency International 2013: 26 of 100 points, rank no. 140 worldwide. Health: Corruption Perception Index, some older data available37 38	Rather general; not helpful for policy assessment
Use of one purchaser or many	Two purchasers: MHSD and oblast health authorities	A political one-off decision; no need to measure
Financing
Total expenditure on health	As share of GDP: 4.2% (2012)*; 3.5% (2011), 3.8% (2012), 3.5% (2013)—NHA by MHSD and RCHD	To be included; specific target needs to be set
GGHE as a proportion of GGE	GGHE/GGE: 10.9 (2012)*; GGHE as % of GGE 12.1% (2011), 12.3% (2012), 11.6% (2013) –NHA by MHSD and RCHD	To be included; specific target needs to be set
The ratio of household OOP payments for health to total expenditure on health	OOP as share of THE: 41.7%(2012)* 28.2% (2011), 30.2% (2012), 32.7% (2013) –NHA by MHSD and RCHD	To be included; specific target needs to be set
Health Workforce
Health workers/100 000 population	Physicians: 349.5; Dentists: 41.2; Pharmacists: 80.6; Nurses: 804.5; Midwives: 57.73 (2012)†	To be included; specific targets need to be set
Distribution of health workers by occupation/specialisation, region	GPs: 31/100‘000 (2011 WHO HFA-DB)	Data available from some oblasts (regions); collection and publishing of this data should be consolidated. To be included, specific targets need to be set
Graduates of health professions institutions/ 100 000 population (level/field of education)	Physicians graduated/100 000: 21.86 (2010)†	Quantity is not a key concern; more important is quality, thus not included
Health Information Systems
Health information system performance index	Not available	The WHO recommends compiling the index based on: Health surveys Birth/death registration Census Health facility reporting Health resource tracking Capacity for analysis To be included; specific targets need to be set
Service delivery
General Service readiness score for health facilities	Countrywide assessment has been initiated but results are not yet available	The WHO recommends Health Facility Assessment, including: Basic amenities Basic equipment Standard precautions for infections prevention Laboratory Medicines and commodities To be included; specific target needs to be set
Proportion of health facilities offering specific services(/10 000 population)	ditto
Specific-services readiness score for health facilities	ditto
Service quality
Accreditation and external quality management	For providing the SGBP, a health facility must pass accreditation. Accreditation by the Committee for Control of Pharmaceutical and Medical Activities (MHSD), recognised by ISQua Standards for Facilities	To be included; specific target needs to be set
Involving public/private providers	At present, limited involvement of private providers via state budget; 811 public and 428 private providers	Unspecific indicator; no optimal level defined
Personnel (qualification, number); staffing index/ratio; competency index	Countrywide assessment has been initiated but results are not yet available	In principle, helpful to monitor availability and quality of staff
Utilisation
Use of specific preventive measures	Data currently not available	Already included
Benefit catalogue (needs, economic evaluation)	State-guaranteed benefit package (SGBP) very comprehensive, but not yet evaluated	No need for monitoring
Support value (% of total cost paid)	Not applicable	Relevant intermediate indicator, especially with the planned SHI (financial protection)
Benefit Incidence Analysis (BIA)	Household survey has been conducted, but so far not fully analysed	To be included; specific target needs to be set
Accessibility (e.g. geographical; no. of providers; time to get to a provider for Caesarean section)	Household survey has been conducted, but so far not fully analysed	Relevant intermediate indicator for planning
Access to essential medicines
Average availability of selected essential medicines in public and private health facilities	Median availability (private): 70% Median availability (public): 0% (2004)*	10-year-old data for drug prices and availability; a new survey is required to determine whether performance is still this poor. To be included; specific target needs to be set
Median consumer price ratio of selected essential medicines in public and private facilities	Price ratio (private): 3.73 Price ratio (public): 4.84 (2004)*
Health level
Risk factor prevalence		To be included; specific target needs to be set
Average life expectancy	69.6 years39	Already included
Maternal mortality	26/100 000 live-births39	Already included
Infant mortality	14.6/1000 live-births39	Already included
Overall mortality		Analysis for different ethnicities40 41, regions42 and age groups43 available. Already included
Decrease in specific morbidity, eg, TB, HIV/AIDS		Available from the WHO‡ Already included
Financial protection and equity
Impoverishment rate due to healthcare expenditures	Household survey has been conducted, but so far not fully analysed	Stratified by wealth quintile, place of residence and gender To be included; specific target needs to be set
Kakwani index	Data available for calculation	None
Responsiveness
Customer Satisfaction Rating (facility)	Household survey has been conducted, but so far not fully analysed	To be included; specific target needs to be set
Overall satisfaction with health services	Household survey has been conducted, but so far not fully analysed	To be included; specific target needs to be set. Data from 2001 and 2010 available44

*Global Health Observatory Data Repository; http://apps.who.int/gho/data/node.main.488?lang=en.

†European health for all database (HFA-DB), WHO Regional Office for Europe, http://data.euro.who.int/hfadb/.

‡http://www.who.int/tb/country/en/.

GDP, Gross Domestic Product; GGE, General Government Expenditure; GGHE, General Government Expenditure on Health; MHSD, Ministry of Health and Social Development; NHA, National Health Accounts; RCHD, Republican Centre for Health Development; SGBP, State Guaranteed Benefit Package; SHI, social health insurance.

Discussion

This case study indicates that the standard WHO/World Bank/Global Fund health policy monitoring framework needs to be tailored to the country context. At the same time, the health-related indicators presented in the Kazakh 2020 Strategy have to be refined to attain consistency in quality, usefulness and measurability. On the basis of these findings, a number of indicators, quantitative and qualitative, that would fulfil SMART criteria (specific, measurable, action-oriented, realistic, timely) can be selected. In some instances, perception-based qualitative data would be most appropriate. This mix of approaches and methods would provide as broad a picture as possible. In addition, equity measurements comparing the data for the richest versus the poorest 20% of the population could be incorporated. Roberts et al,²⁹ in a similar vein, proposed a ‘step pyramid’ that allows one to disaggregate the UHC cube.

A simplified WHO/World Bank/Global Fund health policy monitoring framework would assist in examining the effects of Kazakhstan's ambitious health policy reforms (figure 4).

Figure 4.

The modified WHO health system monitoring tool: Assessing inputs and processes, outputs, outcomes and impact. Source: Authors.

This approach incorporates the WHO health system framework, but differs from the WHO/World Bank/Global Fund common evaluation concept in three important points as (1) it does not primarily focus on international comparability, but rather looks at monitoring for the needs of the country, (2) it takes a broader view on the key input parameters (like staff, IT, governance) rather than the composite measures of service coverage and financial protection and (3) it views health equity as a cross-cutting indicator that should be looked at throughout the delivery chain rather than assessing equity only at the outcomes/impact level. Ambiguous indicators, for which it would be difficult to define specific targets, were left out. This framework can then be reapplied to the health-related Strategic Development Goals 2020 in Kazakhstan to provide a revised monitoring concept (see table 5).

Table 5.

A revised monitoring concept for the health-related Strategic Development Goals 2020 in Kazakhstan

ALOS, average length of stay; GGE, General Government Expenditure; GGHE, General Government Health Expenditure; OOP, out of packet payments; SES, socioeconomic strata; SGBP, state guaranteed benefit package.

In addition to strengthening/implementing the General and Specific Service Readiness Scores and the Health Information System Performance Index, a comprehensive household survey would allow for more precise health expenditure and utilisation data, stratified by the wealth quintile, place of residence, ethnicity and gender, which in turn could guide health policy when it comes to specifically addressing equity issues.

As for measuring, estimating and monitoring risk factors, the WHO's STEPwise approach to non-communicable disease risk factor surveillance (STEPS, see http://www.who.int/chp/steps/riskfactor/en/) is a well-established instrument for obtaining core data on risk factors that determine the burden of disease. The STEPS instrument includes a questionnaire as well as physical and biochemical measurements and provides concrete advice on surveillance planning, adapting and implementing the instrument.

Owing to its simple structure and comprehensive approach, the adopted WHO/World Bank/Global Fund health system monitoring tool (see figure 4) seems suitable to determine whether reforms are aimed at the key system features needed for the provision of quality healthcare. However, such a monitoring concept has so far not been used to evaluate reform agendas with respect to their coverage of essential health system features. This novel approach derived from the Kazakh experience should be tried and tested in other contexts as there is a clear gap between policy initiatives and measuring the effects they have.

It is both a scientific and a political question to define what constitutes an improvement and what goals should ideally be achieved by which means. The proposed framework should thus be viewed as a suggestion and stimulus for health policy debate in order to support policymakers and administrators in working for better health, more equitable health and better financial protection while taking into account the non-medical needs of the population.

Conclusion

Kazakhstan's goal to provide universal healthcare coverage for its population is laudable and much needed, but challenging to achieve. While reform efforts have been under way for some time, overall population health indicators still show significant room for improvement. A key philosophy that Kazakhstan hopes to put into action is the concept of ‘shared responsibility for health’. The current plans for imbedding mandatory SHI, in a primarily tax-funded health system, would formally demonstrate the ideal of this concept (although a tax-based system is also based on the idea of solidarity). At the same time, the proposed SHI reform could potentially tackle many of the current challenges faced by Kazakhstan's health system. It might stimulate an increase in total health expenditure, bringing it closer to levels required to meet Kazakhstan's healthcare objectives. It may also prove to be a major step forward in improving the purchase and provision of healthcare under the general notion of a shared responsibility for health. A stand-alone health financing reform, however, will not resolve the numerous challenges in the sector. The Government will need to assess and review the service organisation and delivery practices and, in some cases, build on the best practices of the ‘Semashko’ system simultaneously. For example, strengthening the polyclinic model might be a valuable element in improving the provision of care,²³ as well as in building up strong public health systems on the concept of the sanitary-epidemiological system, used throughout the former Soviet Union.³⁰

In this complex endeavour, the inability to measure progress towards policy objectives will seriously hamper visualising and effectively communicating the value and implications of the proposed reforms. The modified WHO/World Bank/Global Fund health system analysis framework could be linked to the country's 2020 Strategic Development Plan to form the basis for a revised monitoring framework including new potential data sources and possible measures for equity.

Making full use of such a framework, however, requires primary collection as well as tailor-made studies and surveys that will require substantial capacity development. If the Government of Kazakhstan wants to assess and publish the impact of health financing reforms, resources need to be clearly designated to fine-tune and use these instruments. Systematic monitoring, assessment and transparency will be key technical aspects for visualising and communicating the implications of health financing reforms with respect to tangible health benefits for the population. This has been successfully implemented in other settings such as Bangladesh.³¹ Since Kazakhstan is a vast country with a partially but not fully developed infrastructure for health information management systems, there is potential to leap forward in embracing information technologies in a way that might go well beyond current uses.³²

Keeping equity in the picture throughout assessing the delivery process, financing and overall results is an important part of health system analysis (see for examplean analysis on the effects of introducing national health insurance in Taiwan³³ and Thailand.³⁴ By setting more concrete overall goals and incremental targets, with clear definitions for success and/or failure, Kazakhstan's government would be able to continuously track progress and bottlenecks in the reform process. Reporting positive changes can help to attain a broad acceptance of health policy initiatives in the country as reported from Georgia.³⁵

Recent international and regional developments (economic slowdown in neighbouring Russia/China and conflict in Ukraine) as well as the drop in global oil and gas prices³⁶ have had a major influence on the country's economy. The GDP has fallen from US$231 billion in 2013 to US$218 billion in 2014,²⁰ and this has led Kazakhstan to revise its growth targets downward as well as reviewing the state budget in order to achieve fiscal consolidation—this is a time when a government's commitment to long-term and people-centred healthcare reform is needed most. A well thought-of monitoring framework and data-driven decision-making could aid in getting the most out of limited resources.

Findings from the Kazakhstan case study, with a proposed monitoring framework tailored to the country’s needs, could also potentially be relevant for many other middle-income countries, as it shows that with an appropriate framework, data could be put together in a policy-relevant manner, gaps could more easily be identified and a quantitative basis could be provided for measuring and assessing the impact of health policy reform. However, indicators can only describe change, not explain change. Consequently, an evaluation approach should aim to learn from the reforms as they are implemented, and feed this back into policy development.