The Impact of the Smiling Sun Health Communication Campaign on the Demand for Family Planning and Antenatal Care Services in Rural Bangladesh with Controls for Program Targeting and Endogenous Message Recall

Abstract

We examine the effectiveness of the Smiling Sun multimedia health communication campaign in encouraging women to use family health services in rural Bangladesh. We control for endogenous program placement and address the potential endogeneity of self-reported campaign exposure in health behavior equations by estimating a set of exposure, contraceptive use, and antenatal care equations by full information maximum likelihood (FIML). Relative to the exposure effect of 3.7 percentage points in the simple model of contraceptive use, the exposure effect in the FIML model is a larger 5.1 percentage points, corresponding to as many as 40,000 additional users of family planning.

1. Introduction

Health communication campaigns are widely used in developing countries to present valuable information about health services and new medical technologies, to alter cultural norms and attitudes, and ultimately to assist people in achieving better health through the use of essential health services and the adoption of healthy lifestyles. Health communication campaigns have been used to combat the HIV/AIDS crisis (Bertrand et al 2006; Cohen, Wu and Farley 2004; Singhal and Rogers 2003; UNAIDS 1999), to promote family planning and encourage smaller family sizes Bertrand and Kincaid 1996; Montgomery and Casterline 1996; Piotrow et al 1997; Rogers 1995), and increasingly to alter unhealthy behaviors such as smoking and over-nutrition (Bala, Strzeszynski and Hey 2005; Sowden and Arblaster 2005).

Evaluating the impact of health communication campaigns in developing countries, however, is complicated by the fact that mass media and other components of health communication campaigns aim for wide, often complete, coverage of target populations, precluding the use of randomized controlled designs. In addition, evaluations that involve cross sectional survey data often rely upon self-reported recall of exposure to the health campaign, introducing the possibility of recall bias or other issues of sample selection. Finally, an accurate assessment of a health communication campaign must control for other supply-side factors of service provision such as drug availability and the presence of trained personnel in community facilities that may also impact the same outcomes that the communication program is attempting to influence. Unfortunately, it is seldom the case that these other program variables are randomly assigned to communities in a country. Further, there is increasing evidence that failure to control for program placement could lead to biased estimates of program impact (see, for example, Rosenzweig and Wolpin, 1986; Gertler and Molyneauz, 1994; Angeles, Guilkey, and Mroz 1998; and Todd, 2008). The consequences of such targeted behavior can be serious, affecting not only measures of program impact but also causing bias in the estimated impact of any other variables that are correlated with these program variables.

From an evaluation perspective, these aspects of communication campaign activities cause obvious complications. While there will generally be at least some individuals not exposed to a health communication program, or at least that will not recall having been exposed, the absence of randomization in program exposure makes it difficult to claim that exposed and unexposed individuals are statistically equivalent, that is that they are identical in all relevant respects except for exposure to the campaign. In fact, the sample of individuals who ultimately are exposed to the program may differ in very substantial ways from the sample of individuals who are not exposed. Many of these differences may be easily observed and measured in population-based surveys – age, education, urban residence, wealth, or access to media. Other differences may be less easily measured – concern over personal health, health competence, or even physical and social access to health services.

Differences in observable factors pose little problem for evaluators. Measures of these observable factors can be included in econometric models as statistical control variables alongside measures of program exposure. In the absence of any other violations of classical regression assumptions, single equation econometric models can be estimated using standard procedures (Guilkey, Hutchinson and Lance 2006). In fact, the vast majority of papers examining the influence of health communication interventions on health behaviors have treated exposure as if it were exogenous, conditional upon a limited set of control variables Agha 2002; Gupta, Katende and Bessinger 2003; Kincaid et al 1996; Kiragu et al 1996; Piotrow et al 1992).

However, differences in unobservable factors at either the individual or community level that are correlated with both the outcome of interest and program exposure require a more nuanced approach, since such correlation can lead to biased and inconsistent estimates of program effectiveness (Wooldridge 2006). A few papers have examined the potential endogeneity of exposure to health communication programs. Hutchinson and Wheeler (2006), for example, estimate a two-equation model for campaign exposure and use of family planning using a bivariate probit model that assumes that the correlation in the unobservables follows a bivariate normal distribution. In their estimations, exposure to any family planning media messages was associated with only a 5.5 percentage point increase in modern contraceptive use in single-equation models but a more sizable 15.7 percentage point increase when appropriate controls were made for endogenous exposure. Hutchinson, Guilkey and Lance (2006) also estimate a two-equation model for exposure to the Smiling Sun campaign and a set of basic reproductive and child health services and find that ρ, the measure of the correlation in the error terms, was statistically significant for most models, indicating that exposure to the Smiling Sun campaign was an endogenous determinant of health services utilization; Importantly, estimates of the marginal effects of the Smiling Sun controlling for endogeneity were roughly triple those for the single-equation models assuming exogeneity. Similarly, Kincaid and Do (2006) use both two-stage least squares and bivariate probit estimations to test for endogenous exposure, finding significant correlation in the error terms in the bivariate probit estimations but no evidence of endogeneity when using the Rivers and Vuong (1988) test for the case with a continuous measure of exposure. From their case, they conclude that simple models that assume exogenous exposure will yield estimates of program effectiveness that are biased downwards, under-estimating the full effect.

The existing papers, however, have several limitations. First, while each controls for endogenous exposure, they examine the effects only of binary measures of exposure, regardless of the mode or intensity of exposure. None examine the simultaneous effects of multiple modes of exposure (e.g., television versus radio versus print media). As noted above, researchers have often estimated a bivariate probit model, which relies on the restrictive assumption that the cross-equation error correlation follows a bivariate normal distribution. This is in contrast to the non-parametric assumption employed here. The exception is Kincaid and Do (2006), who construct an index of exposure based on the number of specific messages recalled by respondents. Nonetheless they dichotomize the recall variable into high and low exposure for use in the bivariate probit analysis.

Another key limitation with the existing papers is that none have controlled for the possible selection bias associated with using a sample of women with a recent pregnancy. Sample selection bias might be present in such a case if pregnancy is related to unmeasured characteristics of health service quality or availability (or non-availability) and or if unmeasured characteristics of a woman affect both whether or not she is exposed to health communication messages and whether or not she is able to limit her likelihood of becoming pregnant.

In this paper, we fill a gap in the existing literature by examining the impact of a multimedia communication campaign under two common evaluation complications: non-random program targeting and self-reported exposure to the communication campaign. Specifically, we look at how self-reported exposure to the Smiling Sun multimedia communication campaign in rural Bangladesh impacts upon women’s use of two primary health care services: modern contraception and antenatal care during pregnancy.

The Smiling Sun communication program, which ran in Bangladesh from 2001 to 2003, was a multi-channel campaign with the objectives of establishing the Smiling Sun symbol, disseminating important health-related messages, and promoting health services in urban and rural areas at Paribarik Shastha Clinics (Family Health Clinics) operated by the NGO Service Delivery Program. The campaign involved a 26-episode television drama serial ‘Eyi Megh Eyi Roudro’ (“Now cloud, now sunshine”), which aired on BTV from August 2001 through March 2002. Each episode was followed by discussion and a quiz show involving over 800,000 participants (BCCP 2002, 2003). The serial drama covered a variety of health topics, such as child health (e.g., signs, symptoms and recommended treatment for Acute Respiratory Infections (ARI) and diarrhea; the schedule of childhood immunizations); reproductive health (e.g., the importance of antenatal care, postnatal care and tetanus toxoid vaccinations); and family planning topics (e.g., birth spacing and delaying marriage and pregnancy, the benefits of female education and long-term contraceptive methods) (BCCP 2002).

In addition to the serial drama, the campaign included television advertisements, radio spots, posters, billboards, press ads in daily newspapers and local publicity efforts. At the community and clinic levels, activities included advocacy, interpersonal communication, group meetings, loudspeaker announcements and rallies. The logo was displayed on all NGO project clinics, satellite clinics, depot holder homes and directional signs. Billboards were also placed along major highways and thoroughfares.

We use data collected roughly at the beginning of the Smiling Sun campaign in 2001 and then again two years later. Questions were asked of women of reproductive age about whether they had seen the Smiling Sun logo and, if so, where they had seen it – in a television drama, in a television advertisement, on the radio, on a billboard, at a signboard at a clinic, or elsewhere.

We evaluate the impact of the Smiling Sun campaign using a Full-Information Maximum Likelihood (FIML) estimation procedure that allows for correlation in the unobservables affecting both self-reported exposure and health behaviors. Importantly, we evaluate the impact of distinct channels of exposure to the Smiling Sun – via the television drama and advertisement, via posters, pamphlets or billboards, or via some other mechanism.

While questions about antenatal care were asked of all women, when we estimate the FIML model for antenatal care, we restrict our sample to women who had a child in the 12 months preceding the survey in order to correspond with the period during which the campaign was on-going. To control for this selectivity, the antenatal FIML model includes an equation that determines whether or not the respondent had a child within the last year. In both the contraceptive choice and antenatal care models, we include in the specification district-level fixed effects to control for program targeting (see, for example, Angeles, Guilkey and Mroz, 1999). Hausman type tests are used to determine whether or not these fixed effects are necessary to control for program targeting. Finally, we use simulations to compare the FIML results to results from simpler single-equation methods that do not address the potential endogeneity of self-reported exposure.

In the next section, we present the econometric model and estimation strategy that we use for determining the effectiveness of the Smiling Sun campaign. We describe the data in the third section, while section 4 presents the results and section 5 concludes.

2. Methodology

We estimate two sets of equations to examine the impact of the Smiling Sun campaign. The ultimate dependent variables in the two sets are the respondent’s choice of current contraceptive method (none, traditional, or modern) and whether or not the respondent had antenatal care for her child, given that her child was 12 months of age or less at the time of the survey. Because the estimation problem is somewhat more complicated for antenatal care, we describe its estimation strategy in some detail and then discuss how it would be modified for contraceptive method choice. The set of equations that must be estimated are as follows:

$$\ln \left[\frac{P(C_{ij}=1)}{P(C_{ij}=0)}\right]=X_{ij}^C{\beta }_C+M_{ij}^C{\lambda }_C+Z_j^C{\alpha }_C+{\mu }_j^C+{\epsilon }_{ij}^C$$ (1)

Where the dependent variable is the log odds that woman i (i=1,2,…,N_j) from community j (j=1,2,…,M) had a child within the last year. The X’s represent individual-level variables such as age and education that may affect the outcome, while M represents a series of three indicator variables for the three types of Smiling Sun messages modes. The Z variables are facility level variables that could influence the outcome related to the quality of facilities within one kilometer of where the respondent lives. Finally the μ and ε are unobserved error terms at the community and individual level respectively. More will be said on them below.

The second equation in the system is actually a set of three equations for the three types of message variables. We only present one equation since they are identical except for the dependent variable:

$$\ln \left[\frac{P(M_{ij}=1)}{P(M_{ij}=0)}\right]=X_{ij}^M{\beta }_M+Z_j^M{\alpha }_C+{\mu }_j^M+{\epsilon }_{ij}^M$$ (2)

Where the dependent variable is the log odds that person i from community j indicated that they had heard or seen the Smiling Sun via one of the message sources and the right-hand-side specification is similar to equation (1). The final equation in the system is whether or not a respondent who has had a child within the last year (i.e. conditional on C_ij = 1) had at least one antenatal care visit:

$$\ln \left[\frac{P(A_{ij}=1)}{P(A_{ij}=0)}\right]=X_{ij}^A{\beta }_A+M_{ij}^A{\lambda }_A+Z_j^A{\alpha }_A+{\mu }_j^A+{\epsilon }_{ij}^A$$ (3)

Where the dependent variable is the log odds that woman i from community j had at least one antenatal care visit for a child born in the last year.

Estimation of the set of equations is complicated by several factors and we discuss each in turn. First, there is strong evidence from several studies (see, for example Rosenzweig and Wolpin 1986; Pitt, Rosenzweig and Gibbons 1993; Gertler and Molyneaux 1994; Angeles, Guilkey and Mroz 1998) that show that health programs and facilities are often targeted to high need communities. The statistical implication of this fact is that the Z or facility variables may be correlated with unobservable fixed characteristics of the communities represented by the μ’s. Three methods have been used to control for program targeting in the articles listed above. First, first differences have been used to remove fixed unobservables. Second, community level dummy variables have been included, and finally, the placement process has been explicitly modeled. Our solution to this problem, given our large sample size, is to include district level dummies for 46 of the 47 districts in our sample. We also tried Thana level dummies for 145 of the 146 Thanas in our sample but found no change in the results.¹ We therefore report results for the more parsimonious district dummy specification. Note that we assume that only unobserved fixed characteristics of the community influence the outcome variables. This seems reasonable given the relatively short time period between the two surveys. However, in the empirical section of the paper we test this assumption by allowing for time varying unobservables at the district level.

The second complication is that message recall is highly likely to be endogenous to having had a child in the last year and whether or not the respondent had antenatal care. A third complication is that the set of respondents who are included in the antenatal care equation are a self selected set of individuals who have had a child in the last year. Our solution to both these problems is to allow the ε’s in the set of equations to be correlated and to estimate the system of equations by full information, maximum likelihood methods. Identification of the set of equations is automatic, given that the set of equations is highly non-linear. However, we do have exclusions restrictions to aid in identification. For example, it is assumed that ownership of a radio and TV affect message recall but do not have direct effects on having a child in the last year and use of antenatal care. In addition, the presence of IEC materials in facilities is assumed to have direct effects on message recall but no direct effect on the other two outcomes. Tests of the validity of our identification strategy are discussed in the results section of the paper.

We do not make specific distributional assumptions about the ε’s but rather use a Heckman and Singer (1984) type discrete factor method where the distribution of the ε’s is estimated along with the other parameters of the model. The specific type of heterogeneity that we use is referred to as non-linear heterogeneity (Mroz, 1999) that allows for very general patterns of error correlations among the ε’s. In a set of Monte Carlo experiments, Mroz found that the discrete factor method worked almost as well as methods that assumed multivariate normally distributed errors when the true data generated process used normally distributed errors. However, when the true data generating process did not use normally distributed errors, estimation methods that assumed normality performed poorly relative to the discrete factor method. Another possible method of estimation that does not require normality assumptions is instrumental variables. Unfortunately, standard instrumental variables methods would not control for sample selectivity in our antenatal care model. Standard instrument variables methods are also typically not used with categorical variables as we have for contraceptive method choice. In addition, maximum likelihood methods tend to be more reliable than instrumental variables in the presence of weak instruments (Staiger and Stock, 1997 and Mroz, 1999).

Estimation for contraceptive method choice involves joint estimation of the equations specified in (2) for the three message variables with a modification of (3) to a multinomial logit dependent variable with three categories.

3. Data

The data for this analysis come from the 2001 RSDP Evaluation Survey and the 2003 Rural NGO Service Delivery Partnership (NSDP) Survey conducted by the Associates for Community and Population Research (ACPR), a Dhaka-based research firm, with technical assistance by the MEASURE Evaluation Project. The principal objective of the surveys was to measure awareness and use of project health services among women aged 10 to 49 years. The surveys were conducted in areas that were reportedly covered by the NSDP project and in proximate, often contiguous, areas that were intended to be similar in demographic and socioeconomic characteristics but not defined as project catchment areas and hence could provide a comparison sample for the purpose of assessing the project’s impact. Unfortunately, these non-project areas were subject to considerable contamination from the media campaign and could not be used as control areas as would be done in a true experimental design. Coverage of the RSDP program was non-random. The program was specifically designed to provide services to areas not covered by existing government clinics and services, but no formal criteria for how program clinics were distributed was ever ascertained other than the absence of government health services. These areas tended to be poorer than previously served areas.

In each survey, a two-stage cluster sampling procedure was employed based on a sampling frame used for a 1998 survey of all project NGO areas. The sample frame consisted of project catchment areas, or project clusters, in which the population of each cluster was defined as the number of eligible married couples of reproductive age. The 1998 sample was stratified by NGO, and within each stratum, clusters were chosen with probability proportional to the number of eligible couples. For every selected cluster, 150 to 350 households were listed, proceeding from the northwest corner of the area. Following that, 35 to 38 households were systematically selected, with the expectation that at least 30 eligible women (ever married aged 10 to 49 years) would be found for interview.

The 2001 and 2003 Surveys used the same sample frame as in 1998 but with the addition of clusters into which the project had expanded since the baseline and the removal of clusters in which the project had ceased operation. The 2001 and 2003 Surveys were intended to be representative of the project nationally and of the six divisions in which the project operated. The sample domains for the 2001 and 2003 Surveys were therefore: project and non-project areas and Dhaka, Chittagong, Sylhet, Khulna, Barisal and Rajshahi divisions. Because the project operated in only a few clusters in Barisal, this domain was combined with Khulna division. The distribution of division populations was used to allocate the number of clusters by NGO for each. Fieldwork occurred between July and September in 2001 and between June and September in 2003.

In 2001, interviews were conducted with 9,625 women in project areas and 1,322 women in non-project areas. In 2003, 7,507 project women and 4,372 non-project women were interviewed. In 2003, no attempt was made to interview women who had been interviewed in 2001. Nonetheless, since the sampling strategy involved returning to some of the same clusters, some of the same women may have been re-interviewed again by chance.

In order to evaluate the effectiveness of the Smiling Sun campaign in encouraging the utilization of health services, we examine the use of antenatal care during pregnancy and the choice of contraceptive method (none/traditional/modern). To gauge awareness of project services, women were asked whether or not they had ever seen the Smiling Sun logo. If they had seen the Smiling Sun logo, they were then asked where they had seen the logo: on television in an advertisement, on television in a drama, on a poster, on a pamphlet/brochure, on a billboard sign, or on a sign at a clinic. Respondents were allowed to report multiple sources for where they had seen the Smiling Sun Logo.

The 2001 survey included a facility survey while the 2003 survey did not. The facility survey collected information from all health facilities (e.g. government hospitals, thana health complexes, family welfare centers, project static clinics or satellite clinics, or other NGO or private clinics) serving a cluster, the types of services offered at each facility (e.g. antenatal care, family planning, child health services, etc.), staffing, availability of essential supplies, drugs, and equipment, and interviews with health workers. Since most of the same communities were visited in 2003, we simply assign facility characteristics from the 2001 data set to observations in the 2003 data set. This, of course, assumes that there has been very little change in the facilities between the two points in time. To the extent that the facilities have improved, our model would measure a conservative impact for the facility level variables. However, in preliminary runs we interacted a survey year dummy variable with all the facility variables in the model and none proved to be significant. They were therefore not included in the final specification.

As mentioned earlier, we use both the 2001 and 2003 data sets. However, we could only use 2003 data that was from the same communities as the 2001 data so that we could link the 2003 respondents to the 2001 facility data.

Descriptive statistics for the analysis sample are given in Table 1. Overall, the sample included 12,754 women from 2001 and 8,718 women from 2003 for whom no missing values were observed for the variables in the statistical model. Of these, approximately 14 percent had a birth in the year preceding the survey, and approximately half (49.1 percent) made at least one antenatal care visit during the pregnancy. Just under half of all women were using a method of contraception at the time of the survey – 41.1 percent using a modern method and 6.7 percent using a traditional method. Only 12.3 percent of women reported having seen the Smiling Sun on television, either in the drama or in advertisements. A similar percentage – 12.1 percent – recalled having seen the Smiling Sun on a poster, in a pamphlet or on a billboard. Just over 1 percent recalled seeing the Smiling Sun elsewhere.

Table 1.

Descriptive Statistics

Variable	Mean	Standard Dev.
Sample Statistics N=21472
Endogenous Variables
At least one antenatal care visit for child one year old or less (N=3000)	0.491	0.500
No use of contraception	0.522	0.500
Use modern contraceptive method	0.411	0.492
Use traditional contraceptive method	0.067	0.250
Have child one year old or less	0.140	0.347
Recall TV drama or advertisement	0.123	0.328
Recall poster/pamphlet/billboard	0.121	0.326
Recall Other	0.012	0.107
Exogenous Variables
Woman age 20 to 24	0.179	0.383
Woman age 25 to 29	0.178	0.383
Woman age 30 to 34	0.169	0.375
Woman age 35 to 39	0.140	0.347
Woman age 40 to 44	0.111	0.314
Woman age 45 to 49	0.072	0.259
Woman has primary education	0.248	0.432
Woman has secondary education	0.180	0.385
Husband has primary education	0.190	0.392
Husband has secondary education	0.243	0.429
Husband has college education	0.020	0.141
Household assets in 2nd quintile	0.223	0.416
Household assets in 3rd quintile	0.202	0.401
Household assets in 4th quintile	0.213	0.409
Household assets in 5th quintile	0.192	0.394
Number of facilities within 1km that ran out of drugs	0.291	0.546
Sum of the number of contraceptive methods available within 1 km	1.318	2.333
Number of family welfare visitors in facilities within 1 km	0.637	1.779
Number of family welfare assistants in facilites within 1 km	0.045	0.275
Indicator for 2003 survey	0.406	0.491
Total staff at facilities within 1 km is between 1and 5	0.178	0.383
Total staff at facilities within 1km is greater than 5	0.116	0.320
Number of facilities within 1km with piped water	0.004	0.055
Number of facilities within 1 km that are very clean	0.009	0.092
Number of facilities within 1 km with antenatal care posters	0.305	0.542
Household has a radio	0.305	0.461
Household has a television	0.142	0.349

4. Results

Estimation results for the antenatal care equations are displayed in Tables 2-5, while the results for contraceptive method choice are displayed in Tables 6-8. Note that we estimate the three exposure equations jointly with the presence of a child aged one or less and the use of antenatal care, and we then re-estimate the three equations with the multinomial logit equations for contraceptive method choice. The sets of estimated coefficients were virtually identical for the two estimations, as would be expected given the large sample size and the fact that these are reduced form equations. Therefore we only present results for the exposure equations estimated jointly with the antenatal care equation.

Table 2.

Summary Statistics for Models With and Without Endogeneity/Heterogeneity Controls for Antenatal Care Model

	Values of the Likelihood Function
Heterogeneity Corrected Model:		−24080.80
Simple Model:		−24111.52
Gain From Heterogeneity Corrections:		30.72
	Number of Parameters:
Heterogeneity Corrected Model:		329
Simple Model:		311
Increase in Parameters Estimated:		18
Estimated Heterogeneity Distributions*
Point	Probability Weight
1	0.175
2	0.288
3	0.422
4	0.075

^*The estimated points of support for the heterogeneity distributions are reported in the tables associated with each of the outcomes that we model.

Table 5.

Results for Recall Poster/Pamphlet/Billboard and Recall Other

	Recall poster/ pamphlet/billboard		Recall Other
Variable	Coef	SE	Coef	SE
Individual Level Variables
Constant	−5.985	0.611	−5.670	0.646
Woman age 20 to 24	0.102	0.078	0.186	0.215
Woman age 25 to 29	0.167	0.080	0.013	0.226
Woman age 30 to 34	−0.051	0.083	−0.145	0.235
Woman age 35 to 39	−0.195	0.091	−0.128	0.248
Woman age 40 to 44	−0.396	0.102	−0.332	0.289
Woman age 45 to 49	−0.402	0.117	−0.277	0.326
Woman has primary education	0.208	0.063	0.348	0.169
Woman has secondary education	0.371	0.080	−0.117	0.225
Husband has primary education	0.070	0.065	0.062	0.184
Husband has secondary education	0.209	0.146	0.187	0.082
Husband has college education	0.473	0.157	0.464	0.440
Household assets in 2nd quintile	−0.255	0.080	0.045	0.229
Household assets in 3rd quintile	−0.076	0.081	0.248	0.231
Household assets in 4th quintile	−0.085	0.087	0.047	0.253
Household assets in 5th quintile	−0.108	0.107	0.278	0.316
Indicator for 2003 survey	0.813	0.062	−0.195	0.142
Number of facilities within 1 km with antenatal care posters	−0.101	0.047	−0.141	0.158
Household has a radio	0.085	0.058	0.177	0.157
Household has a television	−0.023	0.083	−0.129	0.253
District fixed effects omitted
Unobserved Heterogeneity Effects for the Random Effects Model
Point 1 (Normalized to Zero)
Point 2	3.710	0.692	−3.427	1.527
Point 3	3.502	0.629	−0.876	0.504
Point 4	4.669	1.153	−0.275	0.666

Table 6.

Summary Statistics for Models With and Without Endogeneity/Heterogeneity Controls for Contraceptive Method Choice Model

	Values of the Likelihood Function
Heterogeneity Corrected Model:		−31899.26
Simple Model:		−31931.39
Gain From Heterogeneity Corrections:		32.13
	Number of Parameters:
Heterogeneity Corrected Model:		356
Simple Model:		338
Increase in Parameters Estimated:		28
Estimated Heterogeneity Distributions*
Point	Probability Weight
1	0.416
2	0.443
3	0.133
4	0.008

^*The estimated points of support for the heterogeneity distributions are reported in the tables associated with each of the outcomes that we model.

Table 8.

Results for Contraceptive Method Choice: Traditional Method versus No Method

	Simple Model		Random Effects
Variable	Coef	SE	Coef	SE
Individual Level Variables
Constant	−4.080	0.414	−4.332	0.596
Recall TV drama or ad (Endogenous)	0.236	0.094	0.454	0.753
Recall poster/pamphlet/billboard (Endogenous)	−0.052	0.096	−0.475	0.182
Recall other (Endogenous)	−0.063	0.257	0.016	0.335
Woman age 20 to 24	0.327	0.122	0.377	0.149
Woman age 25 to 29	0.812	0.118	0.927	0.147
Woman age 30 to 34	1.157	0.117	1.294	0.150
Woman age 35 to 39	1.461	0.116	1.636	0.154
Woman age 40 to 44	1.393	0.120	1.578	0.161
Woman age 45 to 49	0.994	0.135	1.140	0.174
Woman has primary education	0.313	0.075	0.353	0.087
Woman has secondary education	0.244	0.099	0.278	0.129
Husband has primary education	0.211	0.081	0.232	0.092
Husband has secondary education	0.377	0.082	0.395	0.098
Husband has college education	0.794	0.188	0.778	0.217
Household assets in 2nd quintile	0.134	0.103	0.163	0.122
Household assets in 3rd quintile	0.261	0.103	0.314	0.121
Household assets in 4th quintile	0.157	0.107	0.189	0.127
Household assets in 5th quintile	0.209	0.120	0.222	0.202
Number of facilities within 1km that ran out of drugs	0.012	0.166	0.081	0.196
Indicator for 2003 survey	0.164	0.061	0.173	0.069
Sum of the number of contraceptive methods available within 1 km	−0.002	0.045	−0.021	0.054
Number of family welfare visitorss in facilities within 1 km	0.027	0.023	0.025	0.026
Number of family welfare assistantss in facilities within 1 km	0.026	0.122	0.040	0.138
District fixed effects omitted
Unobserved Heterogeneity Effects for the Random Effects Model
Point 1 (Normalized to Zero)
Point 2			−0.200	1.173
Point 3			−0.033	1.432
Point 4			26.387	2.179

Before proceeding to a discussion of the substantive results, we first discuss statistical issues associated with program placement and identification.

4.1 Endogenous Program Placement

To test for endogenous program placement, we performed preliminary estimations with and without the district dummies in reduced form models for antenatal care, contraceptive method choice and the three exposure variables. For each dependent variable, we then calculated the joint covariance matrix of the two sets of estimated coefficients (with and without the district dummies) so that we could test to see if the inclusion of district level dummies altered the effects of the independent variables (see Mroz, 1987 for details of the method for calculating the joint covariance matrix of the two sets of coefficients). For antenatal care, a joint test of the null hypothesis that all coefficients were the same with and without the district dummies yielded a p value of zero and so the null was strongly rejected. We then conducted a joint test of the null hypothesis that just the facility level variables (e.g. drug availability, staffing, and antenatal IEC poster variables).were the same in estimations with and without the district dummies. The null hypothesis in this test was also rejected with a p value of zero. However, this was mainly due to a much stronger positive effect of the staffing variables since the poster and drugs variables were not significant with or without district dummies.

Similar results were obtained for choice of contraceptive method. A joint test of the null hypothesis that all of the coefficients in the choice of modern method versus no method equation in the models with and without district dummies was rejected with a p value of zero. A joint test of just the facility variables (drug stockouts, total number of contraceptive methods available within one kilometer, number of family welfare visitors (FWVs) within one kilometer, and number of family welfare assistants (FWAs) within one kilometer) also had a p value of zero. In this case, without district level fixed effects, the sign of the drug stockout variable was positive and the number of methods was negative. In contrast, with controls for district fixed effects, the signs of the two variables were reversed, although with rather large standard errors. In both cases, the coefficient for the family welfare visitors variable was positive but became a stronger positive with district level controls. The strong result for the effectiveness of family welfare visitors in increasing modern contraceptive use was unsurprising, as FWVs have come to play a vital role in family planning distribution at family welfare centers and through their satellite clinics in villages served by family welfare centers and NGO clinics (Schuler and Hossain 1998, Rahman et al 1980).

Finally, we performed the same test for the three exposure variables. In all cases, the p value was zero for a test of the null hypothesis that the coefficients were the same across models with and without district level controls. Tests on just the facility variables were a little more mixed. For TV exposure, the p value was zero while it was .10 and .09 for recall of a poster, pamphlet, billboard and other types of recall respectively.

All of the above tests assume that the district level dummy variables have the same effect in both survey years. This seems reasonable given the short period of time between the two surveys. Nevertheless, we added district dummy variables interacted with time to all of the equations to see if any coefficients were altered by the addition of time varying district effects. We then redid all of the tests discussed above except for the exposure to other types of recall. Other types of recall were omitted from this since only 1% of the respondents were in this category. In no case was the p value of any test less than 10% and most were in the 50% range. We therefore concluded that for this short time period, district level fixed effects are all that are needed.

Our overall conclusion is that using district level dummies to control for endogenous program placement is important and models without such controls could be misleading. Even though our primary focus is on the impact of exposure at the individual level on antenatal care and contraceptive method choice, bias in the facility effects will cause bias in the coefficients of all variables correlated with these facility level variables.

4.2 Identification

The discrete factor model that we estimate is highly nonlinear and it is identified without exclusion restrictions (see Mroz, 1999). However, we have three exclusion restrictions in the use of antenatal care and the choice of contraceptive method models (TV and radio ownership plus number of facilities within one kilometer with IEC posters are assumed to have direct effects on exposure and only indirect effects on the two primary outcomes) and three endogenous exposure variables. This means that each model is exactly identified on the basis of exclusion restrictions. We performed several identification tests to validate our assumptions (see Angrist and Pischke, 2009 for a detailed discussion of identification in instrumental variables models). First, we needed to show that these three variables have significant predictive power in the reduced form exposure equations. A joint test of the null hypothesis that these three variables have no effect in the three exposure equations (nine restrictions) yielded a p value of zero indicating strong joint significance. However, as can be seen in Tables 4 and 5, some of the variables are weakly significant especially for the exposure to “other” category. This may not be surprising since only 1% of the respondents replied affirmatively in the “other” category for exposure. However, we kept the variable in the model just for the sake of completeness.

Table 4.

Results for Recall TV Drama or Advertisement

	Child		Recall TV
Variable	Coef	SE	Coef	SE
Individual Level Variables
Constant	−3.844	1.712	−7.516	13.385
Recall TV drama or ad (Endogenous)	−0.362	0.314
Recall poster/pamphlet/billboard (Endogenous)	−0.512	0.500
Recall other (Endogenous)	2.334	0.895
Woman age 20 to 24	−0.552	0.171	−0.120	0.102
Woman age 25 to 29	−1.194	0.234	−0.265	0.106
Woman age 30 to 34	−2.013	0.313	−0.341	0.112
Woman age 35 to 39	−2.868	0.385	−0.532	0.122
Woman age 40 to 44	−4.059	0.444	−0.745	0.131
Woman age 45 to 49	−5.312	0.540	−1.050	0.157
Woman has primary education	−0.144	0.095	0.513	0.079
Woman has secondary education	−0.083	0.131	0.950	0.119
Husband has primary education	−0.048	0.100	0.252	0.084
Husband has secondary education	−0.021	0.104	0.514	0.082
Husband has college education	0.342	0.253	0.908	0.163
Household assets in 2nd quintile	−0.080	0.118	0.212	0.126
Household assets in 3rd quintile	−0.194	0.137	0.347	0.128
Household assets in 4th quintile	−0.353	0.146	0.668	0.138
Household assets in 5th quintile	−0.299	0.179	0.899	0.170
Number of facilities within 1km that ran out of drugs	0.199	0.206
Indicator for 2003 survey	−0.015	0.099	0.019	0.069
Total staff at facilities within 1km is between 1 and 5	0.045	0.185
Total staff at facilities within 1km is greater than 5	−0.084	0.275
Sum of the number of contraceptive methods available within 1 km	−0.017	0.063
Number of family welfare visitorss in facilities within 1 km	0.009	0.036
Number of family welfare assistantss in facilities within 1 km	−0.380	0.182
Number of facilities within 1 km with antenatal care posters			0.152	0.054
Household has a radio			0.013	0.220
Household has a television			2.063	0.220
District fixed effects omitted
Unobserved Heterogeneity Effects for the Random Effects Model
Point 1 (Normalized to Zero)
Point 2	3.469	0.923	4.124	13.443
Point 3	−0.533	1.314	5.035	13.019
Point 4	0.054	1.770	−2.318	15.467

The second identification test takes advantage of the fact that the model is identified without exclusion restrictions and adds the three identifying variables to the structural models for antenatal care and choice of contraceptive method. This is clearly not ideal. However, in Monte Carlo experiments for a similar model, Mroz (1999) found that the discrete factor model produced relatively stable results even when relying solely on nonlinearities to identify the model and so we felt that it was worthwhile to perform this specification test.

We then test to see if these variables have direct effects on the two outcomes (they should not if the exclusion restrictions are valid). In the contraceptive method choice model, the p value for a test of the null hypothesis that the three variables had no direct effects resulted in a p value of .23. A similar test in the antenatal care model resulted in a p value of .49, and so there is strong evidence that the exclusion restrictions are valid for both models since the p values indicate that the null hypotheses are supported by the data.

Our conclusion is that there is strong evidence that both models are identified by exclusion restrictions. In addition, we note that even when we included the identifying variables in the antenatal care and contraceptive method choice equations and then relied solely on the nonlinear nature of the model to provide identification that the coefficient estimates for the exposure variables were quite stable as would be predicted by Mroz’s (1999) Monte Carlo results. This gives us confidence in the robustness of our results.

A final identification concern is identifying the selection equation for the antenatal care equation. This selection equation determines whether or not a respondent had a child in the last year and is thus eligible for inclusion in the use of antenatal care equations. We hypothesize that the sum of the number of contraceptive methods available within one kilometer, the number of family welfare visitors in facilities within 1 kilometer, and the number of family welfare assistants in facilities within 1 kilometer have direct effects on whether or not a woman had a child in the last year but do not have a direct effect on use of antenatal care. Our first test examined the joint significance of these three variables in the selection equation. A test of the null hypothesis that the three variables jointly had no effect resulted in a p value of zero and so the null hypothesis is strongly rejected. We then included the three variables directly in the use of antenatal care equation. In this case, a joint test that the variables had no effect resulted in a p value of .08 and so we have weak evidence that these variables can actually be excluded. However, the nonlinear model is identified without exclusion restrictions and, once again, estimates of the parameters of the antenatal care equation were robust to the exclusion or inclusion of these three variables.

Our overall conclusion from our identification tests is that there is firm evidence that our statistical models are identified and that our results are quite robust. We now turn to the estimation results.

4.3 Estimation Results

Before turning to the substantive results, we first discuss Tables 2 and 6 which present results for the estimated heterogeneity distributions. In both cases, we followed Mroz (1999) and added points of support to the heterogeneity distribution until the increase in the likelihood function was less than the additional estimated number of parameters. In both cases, four points of support were sufficient. The mass points for the heterogeneity distribution are presented in the appropriate tables. A chi squared test of the joint significance of the heterogeneity parameters yields p values of essentially zero for both the antenatal care and the method choice estimations. Thus, we find strong evidence for the endogeneity of the exposure variables in both sets of estimations.

Next, we turn to the principal regression results in Tables 3-5 for antenatal care and Tables 7-8 for contraceptive use. Because we found that the exposure variables were endogenous in the antenatal care and contraceptive equations, we will focus principally on the results for the random effects models, though the results for the simple (uncorrected) models are presented for comparison as well.

Table 3.

Results for Antenatal Care Equation

	Simple Model		Random Effects
Variable	Coef	SE	Coef	SE
Individual Level Variables
Constant	−3.539	0.562	−5.767	2.238
Recall TV drama or ad (Endgoneous)	0.414	0.162	0.306	0.226
Recall poster/pamphlet/billboard (Endogenous)	0.496	0.140	0.446	0.180
Recall other (Endgoenous)	0.526	0.374	0.775	0.326
Woman age 20 to 24	−0.100	0.111	−0.163	0.128
Woman age 25 to 29	−0.060	0.121	−0.160	0.152
Woman age 30 to 34	−0.252	0.145	−0.401	0.188
Woman age 35 to 39	−0.155	0.193	−0.320	0.234
Woman age 40 to 44	−0.390	0.320	−0.581	0.359
Woman age 45 to 49	0.380	0.724	0.187	0.753
Woman has primary education	0.497	0.102	0.513	0.109
Woman has secondary education	0.653	0.133	0.683	0.146
Husband has primary education	0.085	0.112	0.084	0.118
Husband has secondary education	0.339	0.122	0.360	0.129
Husband has college education	1.241	0.406	1.334	0.454
Household assets in 2nd quintile	0.379	0.127	0.394	0.133
Household assets in 3rd quintile	0.476	0.133	0.478	0.139
Household assets in 4th quintile	0.870	0.145	0.889	0.155
Household assets in 5th quintile	0.957	0.169	1.017	0.186
Number of facilities within 1km that ran out of drugs	−0.272	0.166	−0.289	0.176
Indicator for 2003 survey	0.454	0.093	0.479	0.097
Total staff at facilities within 1km is between 1 and 5	0.167	0.176	0.189	0.185
Total staff at facilities within 1km is greater than 5	1.218	0.284	1.292	0.310
District fixed effects omitted
Unobserved Heterogeneity Effects for the Random Effects Model
Point 1 (Normalized to Zero)
Point 2			1.254	1.026
Point 3			−6.991	2.298
Point 4			0.212	2.112

Table 7.

Results for Contraceptive Method Choice: Modern Method versus No Method

	Simple Model		Random Effects
Variable	Coef	SE	Coef	SE
Individual Level Variables
Constant	−1.229	0.196	−1.233	0.240
Recall TV drama or ad (Endogenous)	0.264	0.051	0.528	0.184
Recall poster/pamphlet/billboard (Endogenous)	0.270	0.048	0.260	0.051
Recall other (Endogenous)	0.256	0.138	0.197	0.150
Woman age 20 to 24	0.613	0.054	0.620	0.055
Woman age 25 to 29	0.943	0.055	0.956	0.057
Woman age 30 to 34	1.103	0.057	1.118	0.059
Woman age 35 to 39	1.088	0.060	1.107	0.062
Woman age 40 to 44	0.650	0.065	0.670	0.067
Woman age 45 to 49	0.045	0.077	0.062	0.078
Woman has primary education	0.060	0.040	0.054	0.040
Woman has secondary education	0.033	0.053	0.009	0.056
Husband has primary education	0.140	0.042	0.139	0.043
Husband has secondary education	0.148	0.046	0.137	0.047
Husband has college education	0.333	0.121	0.306	0.124
Household assets in 2nd quintile	0.163	0.050	0.166	0.050
Household assets in 3rd quintile	0.184	0.052	0.185	0.053
Household assets in 4th quintile	0.148	0.054	0.138	0.055
Household assets in 5th quintile	0.067	0.063	0.015	0.072
Number of facilities within 1km that ran out of drugs	−0.075	0.081	−0.075	0.082
Indicator for 2003 survey	0.129	0.033	0.128	0.033
Sum of the number of contraceptive methods available within 1 km	0.027	0.021	0.026	0.022
Number of family welfare visitorss in facilities within 1 km	0.053	0.013	0.053	0.013
Number of family welfare assistantss in facilities within 1 km	0.036	0.067	0.036	0.067
District fixed effects omitted
Unobserved Heterogeneity Effects for the Random Effects Model
Point 1 (Normalized to Zero)
Point 2			−0.357	0.204
Point 3			−0.492	0.335
Point 4			20.488	2.050

First, in both sets of estimations, we found that the coefficient estimates for most of the key control variables were of the hypothesized signs and were statistically significant. For example, relative to women in the lowest socioeconomic quintile, women in all higher socioeconomic quintiles were more likely to use both antenatal care and modern contraception. For the antenatal care equation, the magnitude of the effect increased with higher asset quintiles. This is as hypothesized and provides us with confidence in the estimation results. Similarly, women with higher levels of education were more likely to use antenatal care, an effect that in the case of antenatal care increased with additional education (primary or secondary versus none). In the case of contraceptive use, husband’s education – either primary or secondary relative to none – significantly increased the likelihood that a woman will use antenatal care, even controlling for the effects of the wife’s education and the socioeconomic quintile of the household. In the case of antenatal care, only secondary education mattered for husbands.

Facility characteristics also showed significant impacts on health care utilization. As expected, drug stockouts – i.e., the number of facilities within one kilometer that experienced stockouts - decreased the likelihood that a woman would use antenatal care, while greater access to services – i.e., the total number of staff at facilities within one kilometer is greater than five – increased that likelihood. For contraceptive use, the most important facility characteristic was the number of family welfare visitors in facilities within one kilometer. Other characteristics, such as the presence of family welfare assistants and the number of contraceptive methods available, had no significant effects on contraceptive use.

Importantly, we found that health service utilization showed important temporal effects, as evidenced by the significance of the coefficient for the 2003 year dummy variable in all models.

Next, when comparing the random effects FIML models with the simple models (Tables 3 and 6) that do not control for unobserved heterogeneity affecting both the health outcomes (antenatal care and contraceptive use) and the exposure variables (recall TV, recall poster/pamphlet/billboard and recall other), we found several important results. First, in both the simple model and the random effects model, we found significant program effects, that is, exposure to the Smiling Sun through the TV drama or ad, or through the poster, pamphlet or billboard positively and significantly affected both the use of antenatal care and the use of modern contraception. While the coefficients themselves are not directly comparable, it is important to note that, in the case of the antenatal care equation, the coefficient for exposure to the TV drama was smaller in the unobserved heterogeneity adjusted model than in the simple model, while the situation was reversed in the case of modern contraceptive use. This will be discussed next.

Simulations were run to help quantify the size of the effects of the recall variables on both use of antenatal care and modern contraceptive use. Since the antenatal care simulations are more complicated, we explain how these simulations were done first. Recall that we jointly estimate a selectivity equation for a woman having a child within the last year along with the use of antenatal care equation and three recall equations. In the simulations we use the complete sample of women and their actual characteristics to first determine the probability that the woman would have a child of age one or less. A random draw from a uniform distribution with range zero to one is then used to determine if the woman has a child. Conditional on having a child, we then determine the probability of use of antenatal care for this woman and then average over the sample of women that were simulated to have had a child in the last year.

The impact of recall is examined by first performing the above exercise assuming that all women in the sample had no recall of a message and then determining the average probability of use of antenatal care for the women who are simulated to have a child and then repeating the exercise by assuming every woman in the sample recalled a message. We do this for two recall variables - recall the Smiling Sun TV drama/advertisement and recall seeing the Smiling Sun on a poster, pamphlet or billboard. The “other” recall category was not used since only 1% of the women reported seeing the Smiling Sun elsewhere.

The simulations for use of a modern method were much more straightforward since no initial selectivity equation was involved. In this case, we simply determine the probability of modern contraceptive use for each woman in the sample using her actual characteristics and then average over the sample of women. The impact of recall is determined in the same way in which it was done for antenatal care.

Table 9 presents the results of the simulations. The standard errors in the table were calculated by using parametric bootstrap methods where we used the fact that the maximum likelihood estimators are asymptotically normally distributed. We use the estimated covariance matrix to sample from a multivariate normal distribution centered at the estimated coefficients. The results shown are averaged over 1000 bootstrap replications. In addition to the recall simulations, we also simulate the impact of education so that we can compare the magnitude of the recall impacts to impact of increases in female education.

Table 9.

Simulation Results: Increase in Percentage Use due to Change in Each Variable (Standard Error of Increase in Parentheses)

	ANC		Modern Method
Variable	Simple	Random	Simple	Random
No Recall to Recall TV drama/ad	7.28 (0.21)	4.47 (0.15)	3.71 (0.21)	5.51 (0.26)
No Recall to Recall poster/pamphlet/billboard (Endogenous)	8.20 (0.21)	6.58 (0.14)	4.07 (0.20)	4.44 (0.24)
No Education to Woman has primary education	8.72 (0.21)	7.59 (0.13)	0.25 (0.21)	−0.45 (0.22)
No Education to Woman has secondary education	10.91 (0.21)	10.34 (0.14)	−0.36 (0.23)	−0.76 (0.25)

The simulations provide clear evidence of the need to control for endogenous recall of the Smiling Sun campaign. For example, in the simple estimations, we found that exposure to the Smiling Sun campaign from the television serial drama or advertisement was associated with an increased likelihood of using antenatal care of 7.28 percentage points and of using modern contraception of 3.71 percentage points. In the FIML models, however, the effect of exposure to the Smiling Sun was smaller – only 4.47 percentage points – for antenatal care and larger – 5.51 percentage points – for modern contraceptive use. In other words, failure to control for endogenous recall led to an over-estimate of the Smiling Sun campaign’s effect on antenatal care use by 38.6 percent but an under-estimate of the effect on contraceptive use of 48.5 percent. The directions of bias were identical for recall of the Smiling Sun via poster, pamphlet or billboard.

While it is inherently impossible to determine what components contained in the unobservables influence the size and direction of the bias, it is incumbent upon us to at least speculate as to their sources. In the case of antenatal care, the reasoning seems clear. The sample of women who report having seen the Smiling Sun campaign on television – either in the serial drama or in an advertisement – are likely more efficient producers of health, more sensitive to health messages, or perhaps more health conscious – both for themselves and their unborn children. Not controlling for this additional factor means that the measure of the effect of the Smiling Sun campaign as estimated in the simple models reflects not simply exposure to the campaign but also the underlying productivity, sensitivity or health consciousness.

In the case of modern method use, the direction of bias is harder to explain. One possibility is that the unobservable influence affecting both current contraceptive use and recall of exposure to the Smiling Sun is unmeasured prior contraceptive use. If prior contraceptive use positively affects current contraceptive use but negatively affects watching health messages – perhaps because the women already consider themselves informed – the direction of the bias in simple estimation methods will be to under-estimate the full effect.

In both cases, however, an examination of the magnitudes of the estimated effects of exposure to the Smiling Sun and their standard errors indicates clear statistical differences in the effects of media exposure when simple methods are used relative to methods that more appropriately address the underlying structure of the unobservable factors influencing individual level exposure and behaviors.

5. Conclusion

Our results confirm those found elsewhere that not controlling for endogenous exposure to health communication campaigns in health behavior equations can lead to estimates of health communication campaign effectiveness that are substantially biased. While these small percentage point differences in estimates may seem inconsequential, when factored over populations as large as that of Bangladesh, differences in estimates of effectiveness of a few percentage points can mean differences in the estimates of the numbers of women affected in the thousands, tens of thousands, or even hundreds of thousands. For example, according to project documents, the NGO Service Delivery Program covered approximately 2,201,846 women aged 15 to 49 in 2001. Using our estimates for the use of modern contraception (Table 9), the difference in estimates between simple methods and the FIML is approximately 40,000 additional users of modern contraception attributable to the Smiling Sun campaign.

This has clear implications for estimates of cost-effectiveness of a health communication campaign, which rely on accurate estimates of the numbers of new users of family planning, antenatal care or other health services attributable to the communication campaign. Programmers, policy-makers, or financiers of health programs may very much be swayed into funding health communication programs, or not funding them, by swings in the magnitude of effect as large as those found in our analysis.