What is the quality of the maternal near-miss case reviews in WHO European Region? Cross-sectional study in Armenia, Georgia, Latvia, Republic of Moldova and Uzbekistan

Abstract

Objectives

The maternal near-miss case review (NMCR) cycle is a type of clinical audit aiming at improving quality of maternal healthcare by discussing near-miss cases. In several countries this approach has been introduced and supported by WHO and partners since 2004, but information on the quality of its implementation is missing. This study aimed at evaluating the quality of the NMCR implementation in selected countries within WHO European Region.

Design

Cross-sectional study.

Settings

Twenty-three maternity units in Armenia, Georgia, Latvia, Moldova and Uzbekistan.

Assessment tools

A predefined checklist including 50 items, according to WHO methodology. Quality in the NMCR implementation was defined by summary scores ranging from 0 (totally inappropriate) to 3 (appropriate).

Results

Quality of the NMCR implementation was heterogeneous among different countries, and within the same country. Overall, the first part of the audit cycle (from case identification to case analysis) was fairly well performed (mean score 2.00, 95% CI 1.94 to 2.06), with the exception of the ‘inclusion of users’ views’ (mean score 0.66, 95% CI 0.11 to 1.22), while the second part (developing recommendations, implementing them and ensuring quality) was poorly performed (mean score 0.66, 95% CI 0.11 to 1.22). Each country had at least one champion facility, where quality of the NMCR cycle was acceptable. Quality of the implementation was not associated with its duration. Gaps in implementation were of technical, organisational and attitudinal nature.

Conclusions

Ensuring quality in the NMCR may be difficult but achievable. The high heterogeneity in results within the same country suggests that quality of the NMCR implementation depends, to a large extent, from hospital factors, including staff’s commitment, managerial support and local coordination. Efforts should be put in preventing and mitigating common barriers that hamper successful NMCR implementation.

Strengths and limitations of this study.

• This is the first study reporting on the quality of the hospital-based near-miss case review (NMCR) in Central Asia and Eastern Europe.

• The assessment included five countries within WHO European Region and was based on a predefined checklist, providing the opportunity to evaluate the implementation of the NMCR approach in a standardised manner.

• In three countries facilities included in the evaluation accounted for all facilities implementing the NMCR within in the country. In the remaining two countries, where the NMCR was implemented in more hospitals, facilities were chosen in dialogue with local authorities (non-probability sampling), and not at random; however, criteria used to select facilities included also geographical distribution (ie, so that different regions were represented) and hospital type (ie, different types of hospitals were selected).

Introduction

Ensuring adequate quality of healthcare is a primary objective of WHO Global Strategy for Women’s, Children’s and Adolescent’s Health 2016–2030¹ and of Health 2020, the European strategic framework setting the policy directions for the 53 member states in WHO European Region.² Quality in healthcare is recognised as essential for the health and well-being of the population, and as a basic aspect of human rights.^3–5

Among the different strategies aiming at improving quality of care at maternity services, the facility-based maternal near-miss case review (NMCR) cycle was proposed by WHO in 2004 as a type of clinical audit.^6–8 With respect to mortality audit, the NMCR has the advantage to imply less legal issues, and is therefore perceived as more acceptable by staff. Near-miss cases are defined as a woman who nearly died but survived a complication that occurred during pregnancy, childbirth or within 6 weeks after pregnancy.⁹ In the facility-based NMCR all hospital staff involved in the management of the chosen near-miss case—including obstetricians, midwives, nurses and ancillary staff—get together to discuss and evaluate the care provided against national evidence-based guidelines, local protocols and standards of care. The aim of the case review is to critically discuss local management, procedures and attitudes, and to identify areas that can be further improved.⁹ Actions to improve quality of maternal healthcare are proposed and agreed by hospital staff, and subsequently monitored to check their implementation, as for a continuous quality improvement process.⁹ One of the key characteristics of this method is the bottom-up approach, aiming at facilitating local ownership of the process, commitment in implementing the proposed recommendations and team building. Currently, the review of severe maternal morbidity cases (‘near-miss’ events) is recommended by WHO as a key action to eliminate avoidable maternal and perinatal mortality and morbidity and improve the quality of care.¹⁰

While in some countries within WHO European Region (such as UK, Norway and the Netherlands) the practice of reviewing maternal near-miss cases was introduced by the government or by professional associations, in several other countries (most often middle-income countries) its implementation was assisted by WHO and/or United Nations Population Fund (UNFPA). In the latter scenario, coverage and quality of the NMCR implementation were usually discussed during workshops,^11–13 but so far they have not been evaluated using a systematic methodology.

In 2015, WHO developed a checklist for assessing the quality of the implementation of the NMCR cycle at hospital level through a systematic methodology.⁹ This study aimed at evaluating the quality of the NMCR implementation in five countries of Eastern Europe and Central Asia, using WHO checklist, to identify common strengths and weaknesses among different settings.

Material and methods

Population and setting

The assessment was conducted in Armenia, Georgia, Latvia, Republic of Moldova and Uzbekistan between June 2015 and October 2016. Countries were chosen based on the following criteria: (1) activities planned by the Ministry of Health (MoH) included a quality assessment of the NMCR; (2) there was a request for technical assistance from WHO or UNFPA.

In all of the countries the NMCR approach was introduced following WHO methodology.⁹ The year of NMCR introduction differed among countries (table 1).

Table 1.

Characteristics of the countries and of the maternity units assessed

	Armenia	Georgia	Latvia	Moldova	Uzbekistan
World Bank classification*	Lower middle income	Upper Middle Income	High income	Lower middle income	Lower middle income
Population (thousands), total†	2969	4358	2060	3514	28 541
GNI per capita, PPP US$†	6990	3280	21 020	3690	1720
Maternal mortality ratio, adjusted†	30	67	34	41	28
Neonatal mortality rate†	10	15	5	9	14
Institutional deliveries as % of total deliveries†	99.4	98.3	NA	99.4	97.3
National introductory workshop on NMCR‡	2007		2012	2005	2005
First national technical workshop on NMCR‡	2009	2015	2013	2005	2007
Number of hospital implementing NMCR‡	3	6	2	13	62
Number of hospital assessed	3	6	2	6	6
Type of hospitals	1 Regional, 2 District	2 Regional, 4 District	1 Regional, 1 District	2 Regional, 4 District	3 Regional, 3 District
Number of births/year in the hospital assessed‡	6125	8570	8152	13 311	23 309

*Source: The World Bank, Country and Lending Groups (2014). Historical classification. Available from: https://datahelpdesk.worldbank.org/knowledgebase/articles/906519 (accessed 9 March 2017).

†Source: Unicef Country statistics (http://www.unicef.org/statistics/index_countrystats.html) (accessed 7 December 2016).

‡Source: WHO mission reports.

GNI, gross national income; NA, not applicable; NMCR, near-miss case review; PPP, per capita.

The number of facilities visited in each country depended on the total number of hospitals implementing the NMCR cycle: in Armenia, Georgia and Latvia all facilities implementing the NMCR were visited; in Moldova and Uzbekistan, where a large number of maternity units are implementing the NMCR, a sample was selected in agreement with the MoH and the national NMCR coordinator/s, following a geographical criteria (ie, so that different regions were represented) and including different types of hospitals. Overall, 23 maternity units were visited in the five selected countries (table 1).

Data collection

Each facility was visited for at least the duration of a whole day by two independent external experts with long-term experience in NMCR implementation. The international team was joined by the national assessors, experienced in NMCR implementation at local level. The team was under the leadership of one international assessor (AB), who participated in all hospital visits, with the objective of ensuring standards procedures in all assessments.

The assessment was carried out using a checklist developed by WHO to evaluate the quality of the NMCR cycle at hospital level (online supplementary table S1). The checklist was developed by WHO in 2014, field tested and optimised for use in early 2015.⁹ The methodology for the quality assessment is fully described in a WHO manual.⁹ Briefly, the checklist includes 50 items, grouped in 11 domains. The sources of information for the assessment include: direct observation and evaluation of one or more NMCR sessions; discussion with participants, coordinators and managers; documents from the NMCR sessions (templates and notes from the sessions); local documents (regional/local policies and guidance documents; protocols and standards for care; documents related to quality assurance, monitoring and supervision; reports on NMCR activities); national documents (national policies and guidance documents, guidelines and reports on NMCR implementation). According to WHO methodology, using WHO manual⁹ as source of standards, each of the 50 items was scored from 0 (totally inappropriate) to 3 (appropriate) (online supplementary table S1). For each of the 11 domains the arithmetic mean and 95% CIs among all the items in that domain were calculated. The median and the range between the first and third quartile (IQR range) were also calculated.

In each facility, immediately after the assessment, feedbacks were discussed with the local staff and plans for improvement of the NMCR implementation were developed, using a simple matrix (online supplementary table S1).

After completing the visits to all maternity units in the country, a national restitution workshop was organised involving representatives from the hospitals, health authorities, professional organisations and partners. During the workshop, achievements and constraints were presented and underlying reasons were discussed. Recommendations for improvement were developed and synthesised in a standard predefined simple matrix (online supplementary table S1).

Ethical considerations

Activities of this observational study were initiated on request of the MoHs and carried out in close collaboration with the health authorities; ethical approval was not required. Information to hospital staff was provided by MoH representatives and local authorities. All people involved in the NMCR sessions were informed about the purpose of the visit and oral consent from the hospital staff and local coordinators and facilitators participating in the observed sessions was obtained. The review of near-miss cases was carried forward anonymously, that is, information that may have disclosed the identity of the patient, or providers of care, was not reported.⁹ This study did not aim at directly comparing countries or single facilities with different background, context and timelines of implementation, therefore, results of the assessment are reported in an anonymous way, according to WHO methodology.⁹ Detailed finding of the assessment together with feedback on how to improve quality of the NMCR implementation were provided to each facility and to each country individually.

Results

The assessment pointed out that quality of the implementation of the NMCR cycle was heterogeneous among different countries, as well as among different hospitals within the same country. Table 2 reports the results of the summary scores, for each of the 11 domains of WHO assessment checklist.

Table 2.

Summary scores

Facilities	Countries																							Mean (95% CI)	Median (IQR range)
	A			B						C		D						E
	H1	H2	H3	H1	H2	H3	H4	H5	H6	H1	H2	H1	H2	H3	H4	H5	H6	H1	H2	H3	H4	H5	H6
1. Internal organisation	1	1	2.5	1	2.1	0.8	2.8	2.3	1.9	3	2	1.7	1.9	1.9	1.6	2.5	0.5	2.9	2.6	2.7	2.3	2.7	2.3	2.0 (1.3 to 2.6)	2.1 (1.7–2.5)
2. Case identification	2.3	1	1.5	2	3	2	3	3	3	3	2.3	2.2	2.5	2.8	3	2	2.1	3	3	3	3	0.7	3	1.7 (1.0 to 2.4)	2.8 (2.0–3.0)
3. Respect of ground rules	1.5	1.5	2.5	1	2	1	3	3	2	3	3	2	1.5	1	1.5	2	1	3	3	3	3	3	3	2.2 (1.4 to 2.9)	2.2 (1.5–3.0)
4. Case presentation	1.6	1.4	2	0.3	2	0.7	2.3	2	0.7	2.5	3	1.8	0.8	2.5	1.7	2.3	1.2	2.3	1.7	1.3	1	2	2	1.7 (1.0 to 2.3)	1.8 (1.1–2.2)
5. Inclusion of users’ views	0	0	0	0.3	1.7	0	3	1.2	0.5	2.5	1.3	0.3	0	2	0	1.4	0	1.8	2.6	2	1.4	1.2	1.2	1.0 (0.1 to 2.0)	1.2 (0.3–1.7)
6. Case analysis	1.5	1	2.5	0.1	1.4	0.3	2	1.6	1.2	2.1	2.6	2.2	0.9	2	1.4	1.3	0.7	2.5	2.8	1.7	1	2.4	1.3	1.5 (0.8 to 2.3)	1.5 (1.1–2.0)
7. Development of recommendations	0.3	1	2	0.1	1.1	0	2	1.8	1.7	1.8	2.6	1.8	0.1	2.3	1	1.9	0.4	3	2.6	1.7	1	2.3	1.3	1.4 (0.6 to 2.3)	1.7 (1.0–1.9)
8. Implementation of recommendations	0	0.5	2	0	0	0	1	1.7	2	2	1.3	0.8	0	3	0.8	2	0.5	3	2.5	1.5	2.5	3	3	1.4 (0.3 to 2.4)	1.5 (0.8–2.3)
9. Follow-up	0	0	1.5	0	0	0	0	0	3	2	2.5	0	0	3	0	1.6	1.3	2.8	1.5	1.5	1.5	2	1.5	1.1 (0.4 to 2.2)	1.5 (0.0–1.9)
10. Documentation and results diffusion	0.3	0.3	2	0.5	1	0.5	2.5	1	2	1.7	1	0.8	0.6	1.5	1.1	0.6	0.3	1.8	2	2.5	2	2.7	1	1.2 (0.5 to 2.0)	1.1 (0.7–1.9)
11. Ensuring quality in the NMCR	0	0	0	NA*	NA	NA	NA	NA	NA	1	1	0	0	0	0	1	0.3	1.5	1.7	1.2	1.2	1.2	1.2	0.6 (0.1 to 1.2)	1.0 (0.1–1.2)

Red, scores between 0.0 and 0.9; yellow, scores between 1.0 and 1.9; green, scores between 2.0 and 3.0.

*In country B piloting started only 6  months before the quality assessment; for this reason the domain 11 was considered not applicable (NA).

NMCR, near-miss case review.

Overall, the first part of the audit cycle (steps 1–6 in table 2, ie, from case identification to case analysis) was on average fairly well performed in all countries (mean score 2.00, 95% CI 1.94 to 2.06), with the exception of the domain ‘inclusion of users’ views’ which was poorly implemented in most facilities (mean score 1.06, 95% CI 0.12 to 2.00). The second part of the audit cycle (steps 7–10), which involves developing appropriate recommendations, implementation of the recommendations, follow-up, documentation and dissemination of results within the facility and the country, was on average poorly performed in all countries (mean score 1.20, 95% CI 0.93 to 1.46). In particular, the domain 11 ‘ensuring quality in the NMCR cycle’, which implies a process of periodical quality assessment, development of recommendation for quality improvement and related actions, was overall substandard (mean score 0.66, 95% CI 0.05 to 1.28), with the exception of country E, where regular monitoring and supervision was carried out by a team that included national and international members.

In each country it was possible to identify at least one ‘champion’ facility, where quality of the NMCR cycle had only minor deficiencies (A-H3, B-H4, C-H1, D-H3, EH1 and H2). On the other hand, in a few facilities (A-H2; B-H1 and H3; DH6) most of the areas assessed were judged as ‘totally inappropriate’.

In some facilities examples of good practices were also observed for domains that were on average implemented on a substandard level at a country level. For example, despite inclusion of users’ views being substandard in most facilities in countries B and D (mean scores 1.11, 95% CI 0 to 2.22 and 0.61, 95% CI 0 to 1.48, respectively) single facilities reached good scores (B-H4 had a score of 3 and D-H3 had a score of 2), being able to regularly interview women and incorporating their views in the development of recommendations to improve hospital care (table 2).

On average, quality of the implementation of NMCR was on a higher level in country E, where evaluation scores pointed out that there were only few weaknesses in implementation compared with other countries (mean score 2.12, 95% CI 1.84 to 2.39).

Table 3 summarises main common strengths and weaknesses in the quality of the NMCR implementation, as divided in three categories: (1) those mostly related to technical aspects, (2) those predominantly of organisational nature and (3) those related to the attitude towards the NMCR. The main technical strength was that, beside the existence of appropriate technical skills in the methodology, most facilities developed several recommendations that were achievable, realistic, time-bound and with a potential impact on the quality of care. Although recommendations were not always well documented (thus resulting in low scores under domain 10), gaps in reporting results did not always indicated actual gaps in implementation, and in many cases several recommendations were actually implemented. This was a common observation in country B, where recommendations were poorly recorded, but several actions to improve quality of care—such as setting up emergency kits and related protocols, and introducing the Modified Obstetric Early Warning Score chart¹⁴—were actually implemented. Among strengths in organisational aspects, the most important was that NMCR was regularly held, and staffing at all levels, including midwives, participated. Main strengths in attitude included the endorsement and application of the basic principles of the NMCR (confidentiality, openness, respecting diverting opinions and avoiding blame).

Table 3.

Strengths and weaknesses observed in the quality of the NMCR implementation

	Strengths	Weaknesses
Technical	In all countries: Technical skills on performing NMCR were on average fair Local protocols were on average present and used Recommendations were usually developed, with several SMART15 characteristics (achievable, realistic and time-bound) Especially in country E: Most maternity teams were able to analyse efficiently a NM case, and to develop relevant recommendations to improve quality and organisation of care and follow-up their implementation	Case definition not complying with national definition Lack of existence and use of local protocols for case analysis Some lack of knowledge and skills in NMCR methodology Case summary, case reconstruction door-to-door, case analysis (including getting to the real point, and "what we did good", and identifications of the underlying reasons using the ‘why-but-why’) not performed well performed in all facilities Recommendations not fully SMART (often not specific nor measurable)
Organisation	In all countries: Staffing at all levels (including midwives and nurses) was involved and in some cases encouraged by facilitator to actively participate in the review process Session participants were mostly those involved in care provision of the case reviewed, and, generally, felt free to ask questions and express their opinions NMCR mostly happened on a regular basis Especially in country E: An excellent national plan for implementation was developed Appropriate normative regulations were developed through regular NMCR sessions By 2015, 90% of maternity facilities were trained and implementing NMCR Regional NMCR coordinators were established There was sustained support from MoH; WHO and partners (also in country C)	Lack of local written procedure for NMCR Irregular meetings in some facilities Lack of involvement of staffing who managed the case Lack of a regional/national coordination and/or continuity in facilitator/coordinator role and/or support from them Lack of trained interviewers Absence of local leaders Lack of support from hospital manager in organisation of the NMCR and in the implementation of the recommendation Lack of follow-up on previous recommendations Lack of production, dissemination and discussion of results of the NMCR cycle Lack of periodical evaluations of the quality of the NMCR When evaluations of the quality was performed, no mechanism ensured that resulting recommendations were taken up
Attitude	In all countries Basic BTN principles were respected in most facilities, including confidentiality Multidisciplinary approach to case reviews was evident in most facilities Managers offered substantial support to organisation of NMCR sessions and implementation of recommendations Staff found this method useful to improve quality and organisation of care Midwives role as participants, but also as coordinators and facilitators Interviews became a routine in most facilities (in particular in country C) Especially in country E: Facilitators succeeded to create and maintain an open and non-threatening environment during sessions; staff felt free to put forward (or ask) questions and express their opinions (also country C) The point of view of women was always collected and presented; some interviews were of excellent quality (also country C) Professionals were praised in case of good care	In some cases lack of respect of other people’s opinion, persistence of blaming, persistence of a wrong attitude that suggested ‘judging others’, rather than moving towards thinking ‘the review is about us’ Lack of active participation in the discussion Insufficient involvement of mid-level staffing Lack of the interviews with woman in some facilities Even where the interview was collected, women’s view were not taken into account when recommendations were implemented Staff not always praised when quality and appropriate care given Staff considers developing recommendations a mere formality, they were not eager to implement them, and take on the role and the responsibility to change practice. Persistence of a system that advocates punishment in some facilities

BTN, beyond the numbers; MoH, Ministry of health; NM, near miss; NMCR, near-miss case review; SMART, specific, measurable, achievable, realistic and time-bound.

Main gaps in technical aspects were: inappropriate case reconstruction, case analysis not getting to the ‘real point’ and not using a ‘why-but-why’ approach (ie, discussion of underlying causes) and recommendations not being fully SMART (specific, measurable, achievable, realistic, and time-bound).¹⁵ Main gaps of organisational nature were: lack of continuity in the role of facilitator/coordinator, lack of proper dissemination of the results (ie, circulation of information within the facility level and at national level on how many and what type of recommendations were developed) and lack of follow-up on previous recommendations. Major gaps in adopting the background philosophy and principles of the NMCR were observed in some facilities such as: lack of respect for other people’s opinion, persistence of blaming and judging others rather than using the NMCR cycle to discuss and improve ways of working and insufficient involvement of mid-level staff. Lack of inclusion of the users’ view, which was a frequent observation, was reported to be due to the lack of trained interviewers, and this was interpreted as not merely an organisational gap, but also as a problem in attitude of the health providers, that is, lack of understanding the importance of taking into account the women’s point of view. Finally, common to most facilities, there was insufficient monitoring and evaluation, and lack of a quality assurance mechanism. In most cases this was due to deficiencies in establishing and efficiently running a NMCR coordination system at national level.

Recommendations developed by local stakeholders during the national restitution workshops were setting specific. Nevertheless, there were several similarities. The most frequent/relevant recommendations developed for implementation at different levels—hospital level, national level, WHO and development partners—are reported in table 4.

Table 4.

Recommendations made by local stakeholders on how to improve NMCR quality

Hospital level	Ensure managerial support for the organisation of the NMCR and for the implementation of the resulting recommendations Aim at regular sessions Ensure active participation of all staff involved in case management, including mid-level staffing Ensure that ground rules are respected Ensure that the review follows the steps suggested in WHO manual9 Ensure that user’s views are collected and taken into consideration Ensure that recommendations developed are SMART Ensure that every session starts by following up on the previous recommendations Document the implementation of the recommendations (provide date and description) Document, analyse and disseminate results of the NMCR at hospital level, including type of recommendations developed and percentage of those implemented
National level	Set up/strengthen the national coordinating team Develop a plan for regular quality assessment and reinforcement Strengthen technical skills among staffing on the principles, methods and practices of the NMCR cycle Practical training on how to conduct interviews in order to collect women’s views Support networking activities among facilities (eg, exchange visits) Document, analyse and disseminate results of the NMCR at national level
WHO and other development partners	Ensure regular and timely technical support for capacity development, including developing skills for women interviews Provide support for developing legal framework and national guidance manual for NMCR Support regular monitoring of the implementation in a coordinated manner Support results dissemination and discussion Support timely quality assessments and subsequent actions for quality improvement Support networking activities among facilities/countries with the objective of improve quality of NMCR cycle Ensure continuous support for updating key national guidelines, local protocols and standards for clinical practice

NMCR, near-miss case review; SMART, specific, measurable, achievable, realistic and time-bound.¹⁵

Examples of the observed impact of the NMCR on quality of care at facility level are reported in online supplementary table S2. Despite progress was often poorly reported both in the hospital and in national reports, several achievements could be observed. These included improved use of national clinical guidelines, development and use of local protocols and standards of care, better availability and organisation of emergency services, improved autonomy of midwives and positive dynamics such as improved team working.

Discussion

This study aimed at evaluating the quality of the NMCR at hospital level in selected countries within WHO European Region using a standardised checklist and methodology. Overall the assessment pointed out that the practise of reviewing near-miss cases at hospital level is currently ongoing in all countries included in this study; however, both coverage and quality of the implementation of the NMCR cycle are heterogeneous. Overall, while the first part of the audit cycle (from case identification to case analysis) was fairly well performed, with the exception of the ‘inclusion of users’ views’, the second part of the audit cycle (developing recommendations, implementing them and ensuring quality) was in general poorly performed. Gaps in implementation were both of technical, organisational and attitudinal nature.

These findings are not entirely surprising. Previous, although less systematic, evaluations in the same geographical area pointed a series of challenges7 8 11 12 in effectively implementing the review of near-miss cases at facility level. Beside technical and organisational challenges, the successful implementation of clinical audits such as the NMCR often calls for a major change in staff’s attitude.7 8 11 12 In the countries assessed, especially in the ex-Soviet countries, the successful implementation of the NMCR aims at moving away from a ‘traditional’ system of carrying forward clinical audits, where blame and punishment were the routine, subjective judgement were the rule and audit involved only doctors, while midwives, other mid-level staff and service users had no voice.7 8 11 12 The ‘traditional’ audit system mainly resulted in punishing single individuals, rather than at looking to the health system failures and finding solutions at organisational level.7 8 11 12 Changing practices involved building knowledge and skills together with a drastic shift in attitude. Given these substantial constraints, the successful implementation of the NMCR at least in one country (country E) and in several champion maternity units in other countries, must be seen as a positive achievement, proving that NMCR can be successfully implemented in different settings.

This paper reports the quality of the NMCR implementation in middle-income countries (Armenia, Moldova, Uzbekistan are lower middle income countries and Georgia is an upper middle income country), where the NMCR was carried forward with relatively limited resources. Findings of this assessment cannot be generalised to other high-income countries of WHO European Region, such as UK, Norway and the Netherlands, where the practice of reviewing maternal near-miss cases has been institutionalised, with major efforts on creating coordinating mechanisms.^16–18 However, it must be acknowledged that the review of near-miss cases at facility level is still not a routine practice in many European countries. We were unable to identify any study reporting on a standard-based assessment of the quality of the NMCR from any country of WHO European Region.

Interestingly, findings of this study suggest that quality of the implementation of the NMCR cycle is not strictly associated to the duration of the implementation. However, it is also true that adequate time is needed for implementation, and completing a pilot phase in a country cannot take <18–24 months from the first technical workshop. In this regard, it must be acknowledged that country B started piloting just 6 months before the quality assessment; therefore, observed results in this country can be interpreted as satisfactory given the short time frame.

The high heterogeneity in results within the same country (such as in the case of countries A, B and D) suggests that quality of the NMCR implementation depends, to a large extent, from hospital factors, including staff’s commitment, managerial support and local coordination. These results are in line with a systematic review on facilitators and barriers to effective implementation of NMCR cycle, pointing out that hospital factors (good leadership), together with a system of coordination (which often includes external support), are key enablers for effective NMCR implementation (Lazzerini, Ciuch, Covi, et al. Submitted, 2017).¹⁹

This assessment pointed out that, despite WHO recommends conducting an interview with the women/her family for each near-miss case, inclusion of women’s view was still substandard in many of the assessed facilities. However, some facilities (B-H4 and D-H3) reached good scores even when this domain was problematic at a country level (table 2). In WHO framework, ‘experience of care’ is one of the two key components of quality of maternal and newborn healthcare, along with ‘provision of care’.^{1 2} The views of women and their families can provide relevant information on aspects related to case management, including important details on what happened, such as organisational issues, communication issues and respectful care. In a study in Moldova, it was observed that the implementation of NMCR improved attitude towards patients,²⁰ while in Kazakhstan it successfully improved patients’ satisfaction.^{21 22}

This study points out that quality in the reporting on the NMCR activities was overall low. WHO manual now provides a series of templates to facilitate a uniform reporting.⁹ Sustained monitoring and evaluation based on appropriate reporting, as well as periodical quality assessments should be part of a strategy to achieve quality in the NMCR implementation.

This paper has the merit of reporting the actual state of implementation of NMCR in a real setting and not in a study setting (where usually a limited number of facilities is involved for a limited period of time, with dedicated human and financial resources). Another strength of the study is that the evaluation was carried out in a systematic way using a predefined standardised tool and methodology, aiming at evaluating all key aspects that contribute to overall NMCR quality (online supplementary table S1).⁹ To our knowledge, no other previous similar systematic evaluations have been performed.

We acknowledge that the scoring system used by the checklist may be open to some subjectivity. However, this scoring system is similar to others extensively used by WHO in the last 15 years for systematic, standard based, quality assessments and it proved to be able to capture key elements of quality of the implementation in both pragmatic and research settings.^23–27 No other validated tool or scoring system exists to assess quality of the NMCR. The checklist and its score system were field tested before use, until when they were considered satisfactory covering all key aspects of quality of NMCR.⁹ The score is attributed by a team of experts, thus reducing subjectivity of the single individual in the evaluation.⁹

As a second limitation, we acknowledge that in two out of the total five countries (Moldova and Uzbekistan), the sample was selected based on MoH indications (non-probability sampling), and one cannot exclude a selection bias towards the better performing institutions. However, we emphasise that the main purpose of the assessment was to create an opportunity at national level, to discuss quality of the NMCR and to develop recommendations for improvement. Subsequent assessments could extend the evaluation to other facilities and monitor progress in specific areas.

Based on the results of this study, in the future, more efforts should be put in evaluating the quality of the implementation of NMCR on a regular basis. More implementation studies should explore interventions aiming at improving quality of the NMCR implementation in different settings.

The objective of this study was not evaluating the impact of the implementation of the NMCR, but rather the quality of the process. Nevertheless, several achievements could be observed (online supplementary table S2), despite this type of information was not consistently available. These results are in line with other studies^28–41 and a systematic review reporting that NMCR is an effective strategy in improving quality of care when measured against predefined standards and it may even significantly reduce maternal mortality in high burden countries (Lazzerini, Richardson, Ciardelli, et al. Submitted, 2017).

Conclusions

Ensuring high quality in the implementation of the NMCR may be difficult in countries of Eastern Europe and Central Asia, but achievable. In the future, more efforts should be put in evaluating the quality of the implementation of NMCR on a regular basis, capitalising from these lessons and preventing and mitigating common barriers that hamper successful implementation. The availability of a new manual on how to implement and to monitor the NMCR at facility level, and of a standard methodology for assessing quality of the NMCR, as well as templates for reporting,⁹ may facilitate this process.