Implementation evaluation of an iron supplementation programme in high-school students: the crosswise model

Abstract

Objective:

To evaluate the implementation of the national health promotion programme (through nutritional education and Fe supplementation) among female high-school students in Iran and to assess the application of the crosswise model (CM) for evaluating the programme’s implementation.

Design:

Two cross-sectional surveys in female high-school students, one using the direct questioning (DQ) method and the other using the CM, were conducted. Two survey questionnaires used to collect the data contained questions about three aspects of the programme: delivering the Fe supplements, consuming them and holding nutritional training sessions.

Setting:

Female high schools, West Azerbaijan Province, Iran.

Participants:

A sample of 2180 students, of whom 1740 were surveyed by the CM and 440 were interviewed by DQ.

Results:

The CM resulted in estimates that were consistently lower than the estimates of the DQ. The proportion of students in the DQ and CM who reported receiving the pills weekly and on a regular basis was 73·2 and 22·5 %, respectively. About 43 and 31 % of students in the DQ and CM surveys reported consuming all pills delivered to them. In the DQ method, only 12 % of students who reported the number of pills consumed had taken the complete dose (sixteen pills).

Conclusions:

Implementation of the Fe supplementation programme in female high schools was poor and insufficient, so that it seems unsuccessful in reducing Fe-deficiency anaemia in adolescent girls. The CM might be considered for evaluating the health programme’s implementation at high schools.

Fe deficiency is the most common form of malnutrition and the most important cause of anaemia worldwide. According to the results of the 2016 Global Burden of Disease study, Fe-deficiency anaemia with 34·7 million years lived with disability was the fourth leading cause of years lived with disability in the world^{( 1)}. In 2010, anaemia affected more than 2 billion people worldwide⁽²⁾; and it is estimated that approximately 50 % of anaemia cases are due to Fe deficiency⁽³⁾. Based on the estimates of WHO, in 2006, the prevalence of anaemia in Iran was 35 % in pre-school children, 33 % in non-pregnant women of reproductive age and 40 % in pregnant women⁽⁴⁾. In several studies, the prevalence of Fe-deficiency anaemia in different age groups of women in Iran was estimated to be about 14–30 %^(5–7). In a recent systematic review, the overall prevalence of Fe deficiency and Fe-deficiency anaemia among the Iranian population under 18 years of age was estimated to be 26·9 and 13·9 %, respectively⁽⁸⁾.

Fe supplementation in high-risk groups is one of the feasible and effective approaches in reducing Fe deficiency⁽⁹⁾. In 2001, Iran’s Ministry of Health designed an integrated Fe-deficiency control programme for groups at high risk of Fe-deficiency anaemia. A national health promotion programme for female high-school students through nutritional education and Fe supplementation was part of this integrated programme. This programme aimed to increase the Fe intake among students by delivering free weekly Fe supplements. Despite more than one decade of the implementation of this programme, few studies have evaluated the programme’s performance. A study in the south-east of Iran showed that 62·3 % of students had taken all the pills delivered to them during the programme implementation period⁽¹⁰⁾. In another study, complete consumption of supplements was about 31 %⁽¹¹⁾. It seems that the programme has been somewhat unsuccessful; a comprehension programme evaluation is warranted.

In surveys, direct questioning (DQ) is the usual method for asking questions. It has been shown that DQ about sensitive issues can influence the participants’ responses. DQ usually has a tendency to under-report socially undesirable behaviours or over-report socially desirable behaviours, which is known as social desirability bias⁽¹²⁾. This kind of bias could also occur in the implementation fidelity evaluation of health programmes by DQ, which leads to distorted answers about the dose of interventions delivered to respondents or their adherence to the programme. Evaluation of health intervention programmes at schools could be subject to this under- or over-reporting. Some students might provide dishonest answers due to fear of being punished by teachers or other school staff, or worrying about being blamed for non-adherence to the intervention. Thus, there will be some degree of overestimation of the intervention delivered to the students. In asking sensitive questions, indirect questioning methods could help obtain more valid estimates. They might increase the confidentiality of answers and protect respondents’ privacy^(13,14). The crosswise model (CM) was introduced by Yu et al. in 2008. It is one of the newest indirect questioning methods⁽¹⁵⁾. It has been used successfully to estimate the prevalence of sensitive issues such as illicit drug use^(16,17), sexual behaviour⁽¹⁸⁾, use of anabolic steroids by bodybuilders⁽¹⁹⁾ and researchers’ misconduct⁽²⁰⁾. However, to the best of our knowledge, this method has not been previously used in assessing the implementation of health programmes.

In the present study, we aimed to: (i) evaluate the implementation of the Fe supplementation programme at senior high schools in West Azerbaijan Province in the north-west of Iran; and (ii) assess the usefulness of the CM for evaluating the health programme’s implementation.

Methods

Iron supplementation programme

Based on a joint agreement between the Ministry of Health and Medical Education and the Ministry of Education, a national health promotion programme through nutritional education and Fe supplementation started in all thirty-one provinces of Iran in 2001. The target population of the programme was female students at senior high schools.

According to the programme guideline, the directory of Population, Family and School Health of the Ministry of Health should arrange sessions to explain the goals of the programme, the importance of the intervention to reduce Fe deficiency, and also to prepare educational materials and deliver Fe supplements at the beginning of each school year. The main intervention was conducted from October to February for 16 weeks. It consisted of three parts: (i) delivering one Fe pill regularly and weekly (totally sixteen pills per student); (ii) consumption of the pills by students under the supervision of teachers; and (iii) holding nutritional training sessions along with delivering the pills. In the case of students’ absence from school, the Fe pills should be delivered to them the next day.

Crosswise model

The CM is a type of non-randomized response technique for sensitive issues and is designed to overcome the disadvantages of previous indirect questioning methods, such as the randomized response technique and its extensions⁽¹³⁾. In this model, each sensitive question (S) is combined with an insensitive question (I). Both questions have a binary response (yes (1)/no (0)). The pair of sensitive and insensitive questions is followed by a joint response with two options, for example ‘A’ and ‘B’. Respondents are asked to give a joint response to the pair of sensitive–insensitive questions. They are asked to choose option ‘A’ if their response to both questions is the same (S = 1 and I = 1, or S = 0 and I = 0), or choose option ‘B’ in the case of one ‘yes’ and one ‘no’ answer (e.g. S = 1 and I = 0, or S = 0 and I = 1). In this way, the options neither specify the existence of sensitive behaviours in the respondent nor dismiss them. This gives a feeling of privacy protection and confidentiality to the respondent, so they can trust the model and avoid false positive or negative responses. The insensitive question should be independent of the sensitive one with known proportion p (e.g. season or month of birth); p must be unequal to 0·5. Using questions with more extreme values of p (i.e. close to, but not equal to 0 or 1) will result in a smaller variance for the proportion of sensitive behaviour. In other words, it means higher efficiency for the prevalence estimation. In the case of p = 0 or p = 1, the model would be the most efficient, but due to a trade-off between statistical efficiency and privacy protection, it would not offer any privacy protection to respondents. Yu et al. argued that working with this model is easy for both respondent and investigator⁽¹⁵⁾. In a recent study, Hoffman et al.⁽²¹⁾ showed that the CM had greater comprehensibility than the other indirect questioning methods, because the respondents simply have to read two questions and integrate their response into one answer option. Some studies showed that the CM provides more valid estimates than DQ^(22,23).

Study design

Female high-school students in West Azerbaijan Province formed our target population. We conducted two cross-sectional studies: one for the DQ and the other for the CM. Both studies were conducted simultaneously, due to logistical problems and the limited time until the end of the school year. We used a multistage stratified cluster-random sampling method to choose the participants from the public and private high schools in urban and rural areas of all seventeen cities in this province.

Sample size

We estimated a sample size of 440 students for the DQ method using P = 0·5, type 1 error = 0·05, 7 % precision, response rate = 90 % and a design effect of 2 for cluster sampling. In comparison to DQ, CM estimates have larger standard error⁽²⁴⁾. To get an adequate level of precision, the sample size of the CM survey must be larger. Thus, we considered a fourfold higher sample size for the CM survey. We selected 2180 students in the form of 109 clusters of twenty students. A sample of 1740 students was surveyed using the CM method. To increase the comparability of the samples, twenty-two clusters of students were randomly assigned to the DQ survey.

Data collection instruments

We used two questionnaires that were made up of two parts. The first part of both questionnaires had the same questions about demographic characteristics. Questions were kept to a minimum number to get a high degree of privacy protection felt by students and contained questions about birth date, school grade and parents’ education.

In the second part of the questionnaires we intended to evaluate three aspects of the programme implementation: delivering Fe pills, consuming them and holding training sessions for students. The second part of the DQ questionnaire was composed of six questions: two questions for each aspect. Each question was followed by a yes/no answer. In addition, we asked the students to report the number of Fe pills consumed.

In the second part of the CM questionnaire, we used the same six questions of the DQ questionnaire, but each question was paired with one insensitive question with a known probability for positive response (Table 1). Each of the paired questions had a joint response with two options: ‘A’ and ‘B’. We used an additional combination of two insensitive questions with known probabilities and joint response. The similarity of the expected probabilities of these insensitive questions with their observed answers could demonstrate that the technique worked properly, and it did not confuse the students. In order to assess the students’ understanding of the instructions and their trust in the technique keeping their information confidential, we added two related questions to the CM questionnaire. The content validity of the questionnaire was evaluated qualitatively by eight academic members. The reliability of the questionnaire was evaluated in a pilot study and its split-half correlation was 0·78.

Table 1.

Questions used in the crossover model technique questionnaire, with the probability of positive answers to the insensitive question

Item	Sensitive question	Insensitive question
1.	Has at least one iron pill been delivered to you by the school officials, during the implementation of the iron supplementation programme?	Take one of your friends or relatives whose birthday you remember. Is his (her) birthday in March? (P = 0·0833)
2.	Has an iron pill been delivered to you weekly and regularly by the school officials, from the beginning of the current school year?	Take one of your friends or relatives whose birthday you remember. Is his (her) birthday between the 1st and 3rd of the month? (P = 0·1)
3.	Have you consumed at least one iron pill you received from school officials during the implementation of the iron supplementation programme?	Take one of the numbers 0 to 9 and do not change it. Is the number 8 your choice? (P = 0·1)
4.	Have you consumed all the iron pills delivered to you by the school officials during the implementation of the iron supplementation programme?	Take the phone number of one of your friends or relatives you know and do not change that. Is the first digit of the phone number 4? (P = 0·1)
5.	Have you received at least one training session about the importance of appropriate nutrition from the school officials during the implementation of iron supplementation programme?	Take one letter of the groups of eight letters from the Persian alphabet and do not change that. Does the selected letter belong to the 2nd group? (P = 0·125)
6.	Have you received regular training sessions from the school officials about the importance of appropriate nutrition simultaneously with the distribution of iron supplements during the implementation of the programme?	Take the cell phone number of one of your friends or relatives you know and do not change that. Is the last digit of the cell phone number 6? (P = 0·1)

P, known probability of the insensitive question.

Data were collected by trained health workers in 2014–2015 school years. The first page of the questionnaire contained instructions to complete the questionnaire. However, oral explanations on how to answer the questions were presented to the students by health workers. Students were assured about the confidentiality of responses.

Statistical analysis

Delivering the Fe supplements and training sessions to the students (which were asked in questions 1, 2, 5 and 6) was considered as ‘intervention delivered to them’. In the DQ survey, to estimate the proportion of participants who had the intervention delivered to them (Pr_DQ; i.e. the proportion of students who gave positive answers to questions 1, 2, 5 and 6), we divided the number of positive answers by the total number of answers for each question (n ₁). The sample variance of the Pr_DQ was estimated using .

In the CM survey, the proportion of participants who had the intervention delivered to them (Pr_CM) was calculated using the following formula:

where p is the known probability of the insensitive question and λ is the proportion of participants who selected option ‘A’. The sample variance of Pr_CM and its standard error were calculated as:

and

where n ₂ is the total number of answers to each question⁽²⁴⁾.

To compare the differences between CM and DQ estimates, a two-tailed Z test for independent samples was calculated using the following formula:

where Z is assumed to have a standard normal distribution.

Results

Table 2 shows characteristics of the participants in the two surveys. No significant differences existed in the proportion of rural and urban students between the two surveys. There were, however, different proportions of students by school type.

Table 2.

Characteristics of the female high-school students (n 2180) in the two surveys (direct questioning (DQ) and crosswise model (CM)), West Azerbaijan Province, Iran, 2014–2015 school years

	DQ survey		CM survey
	n	%	n	%	P value†
Residential area
Urban	259	59·7	1004	58·0
Rural	175	40·3	728	42·0	0·518
Type of school
Private	40	9·1	240	13·8
Public	400	90·9	1500	86·2	0·008

^†P values are based on the χ² test (for proportions).

Table 3 presents the estimated proportions for each of the questions about the Fe supplementation programme in the CM and DQ surveys. The CM estimates were consistently lower than the DQ estimates except for the responses to the question about delivering at least one training session to the students. The largest difference between the estimates of the two methods was observed in the proportion of Fe pills delivered weekly on a regular basis (73·2 % in DQ in contrast to 22·5 % in the CM survey). The proportions of students in the DQ and CM who reported delivery of at least one pill were 99·6 and 78.0 %, respectively. The proportions of all students in the DQ and CM who took all the pills delivered to them were 43·0 and 31·3 %, respectively. Of 364 students who reported the number of pills consumed in the DQ survey, 12 % had taken sixteen pills (complete dose) during the implementation period (results not shown).

Table 3.

Comparison of the results of the crosswise model (CM) and direct questioning (DQ) surveys for different aspects of the iron supplementation programme implementation among female high-school students (n 2180), West Azerbaijan Province, Iran, 2014–2015 school years

	DQ survey			CM survey
	n	%	se	n	%	se	Difference†	se‡
Delivering of at least one Fe pill to students during Fe supplementation programme	440	99·6	0·3	1740	77·9	1·3	21·7**	1·0
Delivering of a weekly Fe pill on a regular basis	440	73·2	2·1	1740	22·5	1·3	50·7**	2·4
Consumption of at least one Fe pill	433	87·9	1·6	1740	76·8	1·4	11·1**	1·8
Consumption of all delivered Fe pills	428	43·0	2·4	1740	31·3	1·4	11·7***	2·6
Delivering at least one training session to students	434	62·5	2·3	1740	63·9	1·6	–1·4*	2·6
Delivering training sessions to students along with the distribution of Fe tablets	354	27·6	2·4	1740	13·8	1·2	13·8**	1·4

^*P = 0·2954,

^**P < 0·001,

^***P = 0·001.

^†The difference between the results of DQ and CM.

^‡Standard error of the difference, estimated by .

No significant differences existed between the expected and observed probabilities of the two insensitive questions that were paired together (Table 4).

Table 4.

Evaluation of the students’ responsiveness and estimates obtained by the crosswise model (CM) survey among female high-school students (n 2180), West Azerbaijan Province, Iran, 2014–2015 school years

	Expected distribution (%)	Observed estimate by CM (%)	se
Were you born in June?	8·3	7·5*	1·1
Were your born between the 1st and 3rd of your birth month? (P = 0·1)	10·0	9·2**	1·0

When the second question presented as insensitive question: *P = 0·205.

When the first question presented as insensitive question: **P = 0·248.

In the CM survey, about 72 % of the students stated that they fully understood the instructions to complete the questionnaire, and 67 % of the students were highly or very highly confident that the CM kept their information confidential. The proportions of the students who had low or very low levels of understanding about the instructions and trusting the CM method were very low (8·3 and 4·7 %, respectively). We found a positive correlation between understanding the technique and trusting the information confidentiality. In other words, students who highly understood the completion instructions, highly trusted the method (Spearman’s ρ= 0·352, P < 0·001; Table 5).

Table 5.

Evaluation of the students’ understanding and trustworthiness of the crosswise model (CM) among female high-school students (n 2180), West Azerbaijan Province, Iran, 2014–2015 school years

		Trustworthiness of CM
		Very low		Low		Moderate		High		Very high		Total
		n	%	n	%	n	%	n	%	n	%	n	%
Understanding of completion guidelines	Very low	7	20·0	5	14·3	15	42·9	8	22·9	0	0·0	35	2·0
	Low	6	12·8	6	12·8	17	36·2	12	25·5	6	12·8	47	2·7
	Moderate	23	5·8	41	10·3	119	30·0	139	35·0	75	18·9	397	22·9
	High	7	1·3	27	5·0	151	27·8	197	36·2	162	29·8	544	31·3
	Very high	1	0·1	21	2·9	121	16·9	193	27·0	378	52·9	714	41·1
Total		44	2·5	100	5·8	423	24·4	549	31·6	621	35·7	1737	100·0

Discussion

In the present study, we evaluated the implementation of the national health promotion programme for female high-school students using the CM and DQ methods. The CM resulted in estimates that were lower than the DQ estimates. In other words, degrees of over-reporting existed in the positive responses to the questions about the implementation of all aspects of the programme in the DQ survey. Our results indicate that all the three major aspects of the programme (delivering and consumption of the supplements, holding nutritional training sessions) were poorly implemented.

We considered evaluating the implementation of the programme as a kind of sensitive issue. Using DQ to estimate the prevalence of sensitive issues resulted in some consequences including wrong answers, item non-responses, over- or under-reporting of the problem and low response rate. Then, in addition to DQ, we evaluated the programme implementation by the CM – an indirect method for asking sensitive questions. It was used successfully in research surveys about highly personal or sensitive questions such as addiction⁽¹⁶⁾, sexual behaviour⁽²⁵⁾, cheating and plagiarism⁽²⁰⁾. A more-is-better hypothesis has been assumed in these studies, and results indicated that the CM method provides a higher prevalence for sensitive behaviour⁽¹⁵⁾. In the present study, we assumed that social desirability bias could lead to inflated and biased estimates of services delivered to the students. Therefore, we assumed that the size of the CM estimates for each item would be lower than the DQ estimates, and we considered a lower-is-better hypothesis. Our findings revealed that DQ can be subject to considerable over-reporting in the evaluation of the health programme at schools. We found significant differences between the results of the DQ and CM methods with regard to the three major aspects of the programme. For example, we found a 51 % difference between the estimates of the DQ and CM method for the proportion of students to whom Fe supplements were delivered weekly and regularly, and a 12 % difference for consuming all the pills delivered to the students (Table 3). Hence, it seems that the proportion of the students who consumed all sixteen pills might be lower than that reported in the DQ survey. These findings confirmed our lower-is-better hypothesis.

The success of the CM depends heavily on two factors: understanding the instructions for completing the questionnaire or the comprehensibility of the method; and the trust in the confidentiality of personal information. In the CM survey, we found a positive relationship between understanding the method and trusting in its privacy protection. A considerable proportion of the students in the CM survey stated that they had highly understood the completion instructions and believed in the confidentiality of their personal information. The proportion of the students who did not understand the completion instructions of the CM questionnaire or did not trust in it, was very low. The joint distribution of the students’ responses to these questions revealed that the students who could not understand the method, poorly trusted in it, and vice versa. This is unlike the results of the study by Hoffmann et al.⁽²¹⁾, who did not find any correlation between these two constructs. In addition, we considered the lack of missing data in the results of the CM, which indicates that distrust of the CM or lack of understanding the instructions by some of the students did not lead to non-response to questions.

In our DQ survey, the proportion of students who received Fe supplements regularly and weekly was slightly lower than the results of the study conducted by Kheirori and Alizadeh (79·3 %) in Tabriz, the capital of East Azerbaijan Province⁽²⁶⁾. This difference may be due to the fact that our study was done in rural and urban schools of all seventeen cities of the province. It is possible that the implementation of the programme in provincial cities and rural areas is different from implementation of it in the capital city of a province. In our DQ survey, furthermore, the proportion of the students who consumed all the Fe pills delivered (43 %) was similar to that in the study conducted by Khammarnia et al.⁽¹¹⁾ in Semnan (38·2 %), but was remarkably lower than that found by Kheirori and Alizadeh (62·3 %) in Tabriz⁽²⁶⁾. The low consumption rates of Fe pills could be attributed to adverse effects of the Fe pills, lack of water supply for taking the pills in class and students’ low levels of health knowledge. The latter may be the results of insufficient nutritional education provided by teachers, the limited number of training sessions, teachers’ unwillingness and reluctance to hold training sessions, and insufficient time to hold them, similar to what was found in other studies in Iran^(11,26,27). The results of our DQ survey revealed that almost one-third of the students received regular educational sessions along with the distribution of pills. The small number of training sessions might have affected the consumption of pills by students. It might explain the very low percentage of students (12 %) who consumed all sixteen pills during the programme implementation period. These findings show that the programme was poorly implemented, and it cannot be successful in achieving its goals.

The present study has some strengths. To the best of our knowledge, it is the first study that used the CM for the purpose of examining the implementation of a health programme. Also, we carried out a DQ survey concurrent with the CM and compared the results. We selected clusters of students randomly for the two methods and believed that we could avoid bias from the time lag between the interviews in our results. We used a larger sample size in the CM method than the DQ method to get the estimates with sufficient precision. Furthermore, according to some authors, asking questions about an issue that may seem repetitive should be avoided⁽¹⁶⁾. It may affect the respondents’ trust. They may think that the researcher tries to find their response by linking questions to each other. To overcome that, we used a limited number of questions for each aspect of the programme, which in turn limited the amount of the information obtained.

Our study has some limitations. First, we did not ask any questions about the comprehension of the DQ survey and trusting in its privacy protection. Instead, we considered the non-responses to some items in the two surveys. Some degree of non-response appeared in the DQ, which means that some students did not trust in the DQ method completely. In an empirical study that compared the DQ method with some indirect questioning methods, Hoffmann et al.⁽²¹⁾ showed greater comprehensibility and lower perceived privacy protection of DQ than indirect questioning methods, including CM. Second, we assumed that social desirability bias and non-response bias could threaten the validity of the programme implementation quality and the estimates obtained through the DQ method, so we used CM to obtain more valid estimates. But we had no valid estimates about the known prevalence of the intervention delivered, or adherence of the students to the intervention, which could call into question the validity of our CM estimates. To overcome this problem, we used the results of the DQ method as a control for the results of the CM method. In addition, we considered the lack of missing data and the high frequency of students who understood and trusted in the CM method. Furthermore, we used a set of two insensitive questions in the CM questionnaire and compared the corresponding estimates with their known probabilities to assess the degree of accuracy in the responses of the students. No significant differences existed. This means students understood the completion instructions and trusted in it, so that their correct answers to these questions could signify valid estimates for the questions about the programme implementation. Also, the results of two recent studies using empirical data showed that the CM method is capable of providing a valid estimate for the prevalence of sensitive behaviour compared with DQ^(22,23). Some studies, nevertheless, showed that using the CM leads to non-ignorable false positives that makes the validity of the estimates of CM questionable^(28,29). These studies, however, used a very rare condition in the population to show the rate of false positive in the estimates. More research is needed on the validity of CM estimates when the condition of interest is not rare, and both false positive and false negative results are possible.

Conclusion

In conclusion, the present results showing poor implementation of all aspects of the programme at schools were in line with the findings of previous studies. The poor implementation quality of the programme and the incomplete and irregular consumption of Fe supplements by the female high-school students make the programme ineffective in reducing both Fe deficiency and Fe-deficiency anaemia in this group. Furthermore, our results showed that the CM might give better estimates about the implementation of health programmes at schools. Much more research is needed, however, both in assessing the validity of the results of the CM, and on the application of this method to evaluate health programmes at schools and in other settings.