Validating the Verbal Autopsy Questionnaire for Maternal Mortality in Pakistan

Abstract

Maternal mortality represents the largest and the most persistent gap between developed and developing countries. The maternal mortality ratio (MMR), which measures the risk of death at each pregnancy, is up to 40 times higher in some African countries than the countries of Northern Europe. MMR is believed to be the most sensitive indicator of women’s status in the society and of the quality and accessibility of maternal health services available to women. The primary purpose of the study was to determine the level of agreement in the cause and the category of death assigned by the hospital and the obstetrician who reviewed the completed VA questionnaire. The validation study was conducted during July–September 2006 by the National Institute of Population Studies (NIPS), which is also the executing agency for PDHS 2007. The primary objective of the study was to test the sensitivity and specificity of the VA questionnaire developed for PDHS 2007. An important lesson learned from this study is that the sensitivity of the VA technique can be further enhanced by rigorous training of the review panelists in the utilization of the information contained in the coded parts of the questionnaire.

Introduction

Maternal mortality represents the largest and the most persistent gap between developed and developing countries. The maternal mortality ratio (MMR), which measures the risk of death at each pregnancy, is up to 40 times higher in some African countries than the countries of Northern Europe ^[1]. MMR is believed to be the most sensitive indicator of women’s status in the society and of the quality and accessibility of maternal health services available to women. A maternal death is not merely a result of treatment failure; rather it is the final outcome of a complex interplay between a myriad of social, cultural and economical factors. Therefore, maternal mortality is widely recognized as a key human rights issue ^[2]. A maternal death is a sign of gross social injustice to women. It implies that the society has failed to look after the life and health of its mothers.

It was for these reasons that the Millennium Declaration adopted MMR as an indicator of maternal health, and set the goal of reducing maternal mortality by 75% (from the 1990 level) before 2015. Many experts believe that achieving this target is theoretically possible in a majority of countries where the only need is to introduce skilled birth attendance and to improve the accessibility and quality of emergency obstetric care ^[3].

Unfortunately, maternal mortality is an area where the least progress has been made since the United Nations’ Millennium Declaration in 2000. All projections, using the trends in reduction of maternal mortality since 1990, point that the 5th Millennium Development Goal (MDG-5) will not be achieved in a vast majority of developing countries. Moreover, an added complexity is that it is hard to measure the MMR, which requires a well-developed births and deaths registration system, or overly expensive field surveys. Indeed, this is the most important reason noted for a failure to address this MDG in the developing nations of the world ^[4].

In the absence of a vital registration system, the levels, medical causes and risk factors of maternal mortality can only be determined through population-based studies. To determine the MMR (and the causes of maternal mortality) at the national level, large-scale surveys are needed.

This paper reports the results from a study to validate the verbal autopsy questionnaire to identify maternal deaths in Pakistan. The verbal autopsy questionnaire was later used by the Pakistan Demographic and Health Survey (PDHS) 2007, an important objective of which was to estimate the MMR from a nationally representative sample. To identify the maternal deaths, the PDHS 2007 used the validated verbal autopsy (VA) questionnaire, which was administered to investigate the causes of female deaths in the age-bracket of 15–49 years. The purpose of the VA questionnaire was to determine the causes of death and to classify female deaths into maternal and non-maternal categories.

It goes without saying that the accuracy and reliability of the MMR estimates from PDHS 2007 will depend entirely upon the sensitivity and specificity of the VA questionnaire. The VA questionnaires for maternal mortality have gradually evolved over last two decades. In Pakistan, different versions of the VA instrument have been used in various small-scale studies. However, none of these the questionnaire was ever validated. The validation process requires comparing the results of the VA questionnaires to a gold standard, such as the cause of death assigned by a doctor, for deaths occurring in a hospital, at the time of death on the death certificate or in the patient’s records.

Methodology

The validation study was conducted during July–September 2006 by the National Institute of Population Studies (NIPS), which is also the executing agency for PDHS 2007. The primary objective of the study was to test the sensitivity and specificity of the VA questionnaire developed for PDHS 2007.

The minimum sample size for this study was estimated to be 128, which was based upon the assumption that at least 25% of female deaths in the 15–49 years age-bracket are due to maternal complications. This assumption was derived from the following information: 1) MMR in Pakistan is believed to be around 500 per 100,000 maternal deaths ^[5]; 2) the crude birth rate (CBR) is estimated at 30 per 1000 population; and 3) the age-specific death rate (ASDR) for females 15–49 years is 3 per 1000 ^[6].

The first step of the study was to identify 128 female deaths in the age-bracket of 15–49 years occurring in the four tertiary care hospitals selected for this study. Only the deaths occurring during last one year were included in the study. The deaths were identified by two methods:

1. Backward tracking (identifying all female deaths occurring in the hospital in the stipulated time period and ascertaining the cause of death, as well as the home address of the deceased woman, through hospital records). Due to incomplete information in hospital records, this method was not much successful.

2. Forward tracking (identifying female deaths in communities residing in the vicinity of the hospitals and then finding the hospital records pertaining to that death). This was the more commonly used method. Female deaths in communities were identified through the Lady Health Workers’ records.

The information collected on each death was as follows: A certificate of cause of death from the hospital authenticated by a staff doctor; home address of the deceased woman, in order to conduct the VA interview with the next of kin of the deceased woman; and the completed VA questionnaire.

The identification of female deaths was the responsibility of two male Filed Supervisors, who also filled out a “certification of death” form with the help of the hospital staff from the hospital records (electronic records or case files). The information on the form included the name, age, address and date of death of the deceased woman; the name and address of the hospital where the death occurred; categorization of death (maternal, non-maternal and unknown); and the immediate and underlying causes of death. These records were kept strictly confidential and the VA interviewers did not have access to them. Once the address of the deceased woman was confirmed, a team comprising two VA interviewers was dispatched there, who filled out the standardized VA questionnaire.

The VA questionnaire included the following sections: 1) personal information of the respondents included in the VA interview (name, age, sex, relationship with the deceased woman and whether he/she was present during the fatal illness, at the time when the woman was taken to hospital, and at the time of death); 2) personal information of the deceased woman, including a short pregnancy history; 3) a detailed verbatim report of the events surrounding the death, as narrated by the respondents; 4) a check-list of signs and symptoms occurring during fatal illness; 5) details of a few selected signs and symptoms identified through the check-list (duration, intensity, persistence, prognosis); and 6) details of health services utilization during fatal illness.

The VA interviews were conducted by two female interviewers in each team (Rawalpindi/Islamabad and Hyderabad, respectively); these interviewers had extensive experience in conducting verbal autopsy interviews. They were also a part of the VA questionnaires pre-testing team previously, and were trained by the author.

The completed VA questionnaires were sent to NIPS where these were edited and entered into a computer database. Each completed VA questionnaire was then forwarded to a panel of obstetricians at the Pakistan Institute of Medical Sciences, Islamabad. The reviewing obstetricians carefully read the completed questionnaire and then filled out a “certification of death” form, which was exactly similar to the one filled at the hospital where the death had occurred. Like the VA interviewers, the reviewing obstetricians were also not privy to the information contained in the hospital’s certification of death form.

A maternal death was defined as “the death of a woman while pregnant or within 42 days of the termination of pregnancy, irrespective of the duration and site of pregnancy, from any cause related to or aggravated by the pregnancy or its management but not from accidental or incidental causes” ^[7]. A “pregnancy-related death”, on the other hand, was defined as “the death of a woman while pregnant or within 42 days of the termination of pregnancy, irrespective of the cause of death or duration and site of pregnancy”.

The primary purpose of the study was to determine the level of agreement in the cause and the category of death assigned by the hospital and the obstetrician who reviewed the completed VA questionnaire. Using hospital diagnosis as the “gold standard” and regarding the VA interview as a screening test, the sensitivity, specificity and the positive and negative predictive values were computed, as follows:

Category of death assigned by the VA review panel	Category of death assigned by the hospital where death occurred (Gold Standard)		Total
	Maternal	Non-maternal
Maternal	A	B	A+B
Non-maternal	C	D	C+D
Total	A+C	B+D	N

Sensitivity = [A / (A+C)] * 100

Specificity = [D / (B+D)] * 100

Positive Predictive Value (PPV) = [A / (A+B)] * 100

Negative Predictive Value (NPV) = [D / (C+D)] * 100

The causes of death assigned by the hospital and the reviewer were compared through contingency tables. The degree of agreement was estimated, using Kappa statistic.

Results

Only 120 female deaths could be identified and successfully tracked in the two sites (Rawalpindi/Islamabad and Hyderabad) and VA questionnaires were completed on all of these deaths. However, 10 questionnaires were discarded due to missing information.

Out of the 110 deaths included in the analysis, 39 were classified as “pregnancy-related”, as it was established through the VA interview that these deaths occurred either during pregnancy or childbirth, or within 42 days of a delivery, miscarriage or abortion.

Table 1 presents the cross-classification of the deaths into maternal and non-maternal categories (using the ICD-10 maternal death definition), assigned respectively by the hospital and the review panel.

Table 1. Classification of female deaths into maternal and non-maternal categories using hospital diagnosis as gold standard.

Category of death assigned by the VA review panel	Category of death assigned by the hospital where death occurred (Gold Standard)		Total
	Maternal	Non-maternal
Maternal	25	7	32
Non-maternal	4	74	78
Total	29	81	110

Accordingly, the sensitivity, specificity, positive predictive value and negative predictive value were estimated as follows:

Sensitivity	86.2%
Specificity	91.4%
Positive predictive value	78.1%
Negative predictive value	94.9%

Moreover, the hospital’s and reviewer’s certificates of the deaths meeting the “pregnancy-related death” definition (ICD-10) were examined to ascertain the categories of death assigned by them. While the hospital regarded only 67% of these deaths as “maternal”, the reviewers assigned 82% of them to the maternal death category.

The hospital’s certification of death classified 25 deaths as maternal. The immediate causes of maternal deaths assigned by the hospital were: Postpartum hemorrhage (8); septicemia (6); hepatitis (3); eclampsia (2); and unknown (6).

The reviewers classified 32 deaths as maternal. The immediate causes of maternal deaths assigned by the reviewers were: Shock due to excessive blood loss (6); postpartum hemorrhage (4); hypertension (4); post-operative complications (3); septicemia (2); other indirect (12); unknown (1).

All deaths were classified into five major cause categories, as follows: Direct maternal deaths; indirect maternal deaths; deaths due to infectious disorders; deaths due to cancers; and deaths due to diabetes and/or hypertension. The deaths where a cause was assigned that did not belong to any of these five classes were classified as the “deaths due to all other causes”. The deaths where no cause was assigned were classified as “deaths due to unknown causes”.

Table 2 presents the distribution of deaths according to the above categories, for causes assigned by the reviewers and the hospital, respectively.

Table 2. Distribution of deaths by cause assigned by the reviewers and hospital^*.

Cause of death	Number (%) of deaths assigned by reviewers	Number (%) of deaths assigned by hospital
Direct maternal deaths	11 (10.0%)	11 (10.0%)
Indirect maternal deaths	16 (14.5%)	10 (9.1%)
Infectious disorders	31 (28.2%)	23 (20.9%)
Cancers	15 (13.6%)	13 (11.8%)
Hypertension/diabetes	28 (25.5%)	12 (10.9%)
All other causes	6 (5.5%)	20 (18.2%)
Unknown causes	3 (2.7%)	21 (19.1%)
Total	110 (100.0%)	110 (100.0%)

^*Kappa statistic for agreement = 0.378 (P = 0.055).

There was complete agreement between the hospital and reviewers with regard to direct maternal deaths. However, the agreement was weak in all other cause categories (the kappa test of agreement gave a P-value of 0.055).

Discussion

In the absence of reliable vital registration systems, a population-based study of maternal mortality is the only remaining option to estimate MMR with reasonable accuracy. The idea of using verbal autopsies to identify maternal deaths through a population-based study is relatively new, and validation studies for the VA questionnaires are scanty ^[8]. Moreover, the inherent qualitative element of the VA interview and the need to interview multiple respondents make it even more difficult to evaluate the reliability and validity of the VA techniques. Finally, completed VA questionnaires need to be reviewed by a panel of experts before a cause of death can be assigned. Each of these steps adds some degree of variation and bias, compromising the quality of information. Nonetheless, the estimation of MMR derived from population-based studies using VA is superior to those derived from indirect methods, e.g. sisterhood method and mathematical modeling ^{[9, 10]}.

We may conclude from this study that the VA questionnaire used by the PDHS 2007 to identify maternal deaths is reasonably sensitive and specific. Even so, it may be noted that the VA questionnaire in its current form will miss approximately 13% of maternal deaths, resulting into significant under-estimation of MMR. Moreover, maternal deaths occurring early in pregnancy might be missed altogether, as noted by other studies ^[1].

A qualitative examination of the completed VA questionnaires, as well as the comments and notes recorded by the hospital physicians and the review panelists on the certification of death forms, was also conducted. It was found that the causes of death reported by the hospital do not follow any standard guidelines, and a wide variety of medical terms are used to describe the same cause in different ways. Unfortunately, the review panelists were no different in this regard. At times, it was difficult to decipher the actual cause of death from the extensive notes on the certification of death forms. Moreover, it seems that the reviewers relied largely upon the verbatim part of the VA questionnaire to arrive at a decision regarding the cause of death, and even for deciding upon the category of death (maternal or non-maternal). There was quite valuable objective information in other parts of the pre-coded questionnaire, such as the check-list of signs and symptoms during fatal illness, which the reviewer did not rigorously consult.

An important lesson learned from this study is that the sensitivity of the VA technique can be further enhanced by rigorous the training of the review panelists in the utilization of the information contained in the coded parts of the questionnaire. A computer-generated summary of the details of the signs and symptoms of the fatal illness, elicited through the objective questioning of respondents by specially trained interviewers, would help the reviewers decipher and understand this information more clearly.

The ideal way to monitor maternal mortality is through a functioning national vital registration system. Until this happens, the only way to reliably estimate MMR will be through verbal autopsies.