Response: Development economics–time to improve research methods

We thank all the reviewers for their comments and insights. Our aim when conducting this appraisal was for the findings of studies to be discussed in the context of their reliability, rather than simply quoted. This discussion is now happening, and readers and policy makers are free to reach their own informed conclusions.

As independent researchers with over 22 years of experience in evidence appraisal, we value transparency in deriving the expected benefits and harms of an intervention; in estimating the size of the effects; and in assessing the level of certainty that these estimates are correct. With community deworming programmes we consider this transparency important for both the children and parents participating in the programmes, and the governments and philanthropists funding them.

A recurring theme of the commentaries is that whereas some of our criticisms stand up and are important, others might be considered over-critical or of limited importance. Nevertheless, when put together we would consider there to be very low certainty in the evidence provided by these three trials.

It seems obvious to us that there is probably too much uncertainty to justify their promotion as educational or economic interventions: the study by Baird et al.¹,² primarily suffers from repeated, iterative analyses which, without a pre-stated primary analysis, are highly susceptible to selective reporting and interpretation. The study by Ozier²,³ has more consistent results in children not dewormed but living in deworming areas, but we struggle with the plausibility of these effects when so many trials of more direct and intensive interventions have failed to demonstrate any effects on intermediate outcomes along the presumed causal pathways. With the study by Croke²,⁴ there is so much uncertainty about the impact of such a large loss of data (over half the original clusters), and the relationship between the 763 included children and the 27 995 children in the original study, that it is probably misleading to consider it a randomized experiment.

The most common defence against criticism of community deworming programmes is that they are simply a more cost-effective way of reaching infected children than a policy of ‘screen and treat’. Although this may be true, both the long-standing Cochrane Review and the recently completed independent Campbell Review conclude that there is no reliable evidence of community-level effects beyond a short-term reduction in the prevalence of intestinal worms.^1,2

We have for some years provided independent critical appraisals and summaries of evidence to help guide policy on a broad range of health interventions relevant to low- and middle-income countries. When we find high quality evidence of consistent benefits, these summaries are welcomed and used by advocates to accelerate the adoption of policies; but when we find the evidence base for existing policies to be weak or inconsistent, the summaries are often attacked or ignored. How the development community now advance with this appraisal we will watch with interest.

More broadly, we hope this debate will strengthen research and analytical methods in development economics, so that the economic and epidemiological disciplines move forward together with a stronger scientific base underpinning the research, its appraisal and whether and how it is used to inform policy.