Measuring the social impact of research

The main aim of health research is to improve the health of people. Yet the performance of researchers tends to be measured by the scientific quality of their research rather than by its impact on health. This is unsatisfactory, even nonsensical, so a committee of the Royal Netherlands Academy of Arts and Sciences is trying to devise a way of measuring the social impact of applied health research. Its first report was discussed at a recent meeting in Amsterdam,¹ and the academy now plans to experiment with methods of measuring social impact.

In an ideal world all research would be of high quality and have considerable social impact by improving health. But in the real world scientific quality and social impact do not always go together. Quality to scientists tends to mean originality of subject, thought, and method. Much research that scientists judge of high quality has no measurable impact on health—often because the lag between the research and any impact may be decades. Thus scientists would think of the original work on apoptosis (programmed cell death) as high quality, but 30 years after it was discovered there has been no measurable impact on health. In contrast, research that is unlikely to be judged as high quality by scientists—say, on the cost effectiveness of different incontinence pads—may have immediate and important social benefits.

The bodies that fund research with public money want both high quality research and social benefit, but current systems for measuring the performance of researchers concentrate on quality. The result might be a serious imbalance in the research portfolio. The Dutch committee concluded that “current methods for evaluation of scientific quality are satisfactory.” In the Netherlands they are based on an assessment made by an international committee of peers using self evaluation by the group being assessed; publications in journals; theses; patents; and plans for the future. Though the Dutch might be satisfied with their methods, there have been strong criticisms of the British research assessment exercise,² and internationally there are doubts about the widespread use of the impact factor of journals for measuring the quality of research.³ Moreover, peer review, a central component of most scientific assessments, has been criticised as slow, expensive, ineffective, biased, prone to abuse, anti-innovatory, and something of a lottery.⁴,⁵

Nevertheless, systems do exist for measuring the scientific quality of research—and are widely used. The Dutch committee concluded that a new instrument was needed for measuring social impact, and that it should be integrated with instruments for measuring scientific quality.

In order to succeed, the Dutch committee said that the instrument should (a) fit with current ways of evaluating research, (b) look to the future also, (c) be efficient for both assessors and the assessed, and (d) work in practice. The committee found that it couldn't produce a “fully worked out methodology,” but it did list indicators that might be used (see box) and advised that criteria to be used for specific assessments should be listed in advance. A message that came through strongly at the Amsterdam meeting—particularly from John Midwinter, a British professor of electrical engineering with long experience of research assessment—was that assessment would work only if the rules were clear. Failing to compare like with like may mean that the assessment will not be fair.

Indicators that might be used to assess the social impact of research

Content analysisProfessional publicationsTreatment guidelines and protocolsPolicy documentsCochrane libraryTextbooksTeaching materialsLay publications

SoftwareCitation analysisScientific publications cited in documents listed above

ProductsHealthcare technologies and servicesInstruments, programmes, methods for assessment or implementation of care

Funding of research(Semi)governmental funding

PublicityPresentations for a non-scientific audienceFact sheetsPublic mediaInternet

MembershipsMember of a committee issuing a policy document or a treatment guidelineMember of advisory committee

TeachingContributions to education of health professionals based on research

The Dutch are now beginning pilots to produce a fully worked out method, but others might also want to experiment with new methods for assessing the social impact of research. There is clearly a need for such a method: otherwise, those who do research with important social benefits may lose out in the increasingly competitive battle for research funds. The ultimate losers would be patients and communities.