Motivating authors to update systematic reviews: practical strategies from a behavioural science perspective

Summary

Systematic reviews are used by clinicians, policy makers and consumers of health care, and should provide up-to-date evidence for clinical decision making. Motivating systematic review authors to update reviews is a challenge, and despite resources such as methodologists and statisticians, many reviews are not updated in a timely manner. Conflicting clinical responsibilities, lack of funding or other essential resources, and waning interest in the review topic are potential barriers to updating. Illustrations of survey questions to elicit updating attitudes and beliefs, and conditions that make updating easier or more difficult within the framework of the Theory of Planned Behaviour, are presented. These surveys may be used to tailor interventions to assist authors with difficulties and may ultimately increase the rate of updates in the published literature.

Introduction

Systematic reviews are the cornerstone of evidencebased health care. Clinicians, policy makers, health care consumers and other decision makers should be assured that reviews are relevant and include the most up-to-date evidence. Within the Cochrane Collaboration, systematic review authors agree to periodically repeat some or all of the steps in the systematic review process and update the contents of the review, when appropriate.¹ This often occurs every 2 years and, if no new evidence is identified, simply requires the author to update the date of search for new evidence. If new studies are identified, authors may be required to review full-text articles, select studies for inclusion, extract data, repeat meta-analyses and summarise findings. Limited time and resources, non-responsive collaborators, and a lack of academic recognition for systematic review updates are potential barriers to completing updates and, for some, a barrier to undertaking a systematic review.^2,3

The methodology of updating reviews is an emerging field of research, and investigators have sought to define the elements of ‘updating,’ estimate how quickly reviews are out-of date, and propose models for when and how to update reviews.^4–7 The success of biennial updates has been limited. In 2005, French and colleagues examined 362 Cochrane systematic reviews initially published in 1998 and found that 30% had not been updated by 2002.⁸ In 1998, Jadad et al. compared the updating status of 36 Cochrane systematic reviews and 39 randomly selected meta-analyses published in 1995 and found that by 1997, 50% of the Cochrane reviews but only 3% of the randomly selected meta-analyses had been updated.⁹ Despite potential barriers, how does one motivate authors to complete review updates 2, 8 and even 10 years after the initial publication?

As a systematic review methodologist and the project director of the Cochrane Eyes and Vision Group (CEVG) US Project, the author works with a group of over 50 systematic review authors. The CEVG, based at the London School of Hygiene and Tropical Medicine, UK, is one of 51 Cochrane review groups that prepare, maintain and promote access to systematic reviews. CEVG review authors prepare reviews of interventions to treat or prevent eye diseases or visual impairment.¹⁰ The CEVG US Project is a satellite based at the Johns Hopkins Bloomberg School of Public Health in Baltimore, MD. It is funded by the National Eye Institute to support US-based ophthalmologists, optometrists and other vision researchers during the systematic review and updating process. The group provides methodological and administrative support and trains vision researchers to perform and interpret the results of systematic reviews. It is from this perspective that the author shares thoughts on motivating authors to update reviews.

The review author' perspective on updating

Motivating authors is an ongoing challenge. Some review authors have access to an editorial team of methodologists, librarians to assist with search strategies and searches, and statisticians but, despite this assistance, still have difficulties in completing their systematic review in a timely manner. Residents, fellows, doctoral students, clinicians, other researchers may have different motivations for undertaking a review. Some conduct reviews to answer a ‘burning question’; others complete a review as a course requirement or as a prerequisite for funding. Some have institutional, private or government funding to conduct the review, and others are unfunded.¹¹ These and other factors may contribute to the author' availability and motivation to complete reviews and updates in a timely manner.

The theory of planned behaviour (TPB) states that the intention to perform a behaviour (i.e. how hard someone is willing to try and how much effort one plans to exert to perform a behaviour) is predicated by: one' attitude towards the behaviour (the negative or positive beliefs about performing the behaviour); subjective norms (one' assessment of the social or other pressure to perform or not perform the behaviour); and (3) perceived behavioural control (how confident one feels about performing the behaviour and how much control one has over the performance of the behaviour).^12–14 The behaviour in this instance is updating a systematic review, or may be more specifically defined by including an appropriate time point for the update (e.g. in 2 years). Developing effective strategies to motivate authors requires an understanding of the review author' experiences during the systematic review updating process, perceptions of potential barriers to completion and beliefs about the utility of updating. So how would one assess intent, attitudes, subjective norms and perceived behavioural control among a sample of systematic review authors?

The intent to update

An assessment of intent should be placed within the context of time. Ajzen suggested multiple questions to assess intent and further suggested that the questions be dispersed throughout an instrument that includes other TPB constructs.¹⁵ An illustration of questions to assess the intent to update is shown below:

• I intend to update my/our systematic review in the next 2 years:

  strongly agree 1 2 3 4 5 6 7 strongly disagree

• I will try to/want to update my/our systematic review in the next 2 years:

  strongly agree 1 2 3 4 5 6 7 strongly disagree

• I plan to update my/our systematic review in the next 2 years:

  strongly agree 1 2 3 4 5 6 7 strongly disagree

Attitudes and beliefs regarding updates

Piehl et al. assessed the barriers to conducting a Cochrane systematic review among a group of 36 lead authors of systematic reviews published in other journals.¹⁶ Thirty-six per cent of the authors responding to the survey expressed an unwillingness to update a review and, further, one author noted a hesitance to commit to an updating policy before determining a long-term research focus.

If new evidence is discovered when updates are performed, some or all of the steps in the systematic review process must be repeated. Depending on the number of additional studies, the amount of time may approach the time required to complete these same steps during the initial preparation of the review. Petrosino surveyed 35 Cochrane systematic review lead authors and found that on average, 28 months were required to complete a review from formulating a question to publication of the review (n = 14, range 18–46 months).¹¹ Allen and Olkin estimated the amount of time required to complete various steps in the systematic review process and found that on average 588 h (51% of total time to complete a review) were required to develop a protocol, search the literature, assess the quality of, extract, and enter data from included studies.¹⁷ Analyses required an additional 144 h, with preparation of the manuscript requiring on average 206 h. Henderson et al. tracked the 2006 status of Cochrane Pregnancy and Childbirth systematic reviews published and pending publication in 1996.¹⁸ Of 86 reviews published or pending publication in 1996, 34 reviews (40%) were out of date in 2006. Of the 34 out-of-date reviews, 76% were not updated because the authors did not have the time to do so.

To date, empirical estimates of the cost of updating are not available, but costs are thought to approach those of the initial systematic review.^19,20 Petrosino found that the median cost of nine Cochrane reviews was US$23 447. If stratified by number of studies included, the median cost of reviews with zero to four studies was US $10 000, while five to nine studies cost $42 578.¹¹ Seventy-one per cent of the 35 lead authors surveyed by Petrosino reported that the most expensive item in the systematic review process was the reviewer' time, with search and retrieval of studies considered to be the second most expensive by 23%.

French and colleagues examined 362 Cochrane systematic reviews initially published in 1998 and found that among the 254 (70%) that had been updated by 2002, 46% did not add new studies and 91% had no change in conclusion. The authors suggest that concerns over fewer updates may not be warranted and further suggest that assessments of priorities for updating may be more appropriate.⁸

Assessing attitudes and beliefs

As suggested by Francis et al.¹⁴ and Ajzen,¹⁵ an assessment of attitudes may include Likert scales for the value of the behaviour (worthless to useful/valuable or disadvantageous/unfavourable to beneficial) and the quality of the experience (unpleasant/unenjoyable to pleasant/enjoyable).

An example of this assessment:

• Performing an update to my/our systematic review in 2 years will be:

  worthless 1 2 3 4 5 6 7 valuable

  unpleasant 1 2 3 4 5 6 7 pleasant

Selecting a representative sample of review authors and conducting open-ended interviews and focus groups to determine a list of advantages and disadvantages to updating may also be used to assess attitudes, with the most frequently mentioned outcomes included in the larger TPB survey. Francis et al.¹⁴ suggested the inclusion of 75% of the beliefs mentioned for appropriate coverage. Examples are:

• If I/we update my/our review in 2 years, I/we will have made an important contribution to the literature:

  unlikely 1 2 3 4 5 6 7 likely

• If I/we update my/our review in 2 years, I/we will waste valuable funding:

  unlikely 1 2 3 4 5 6 7 likely

Assessing subjective norms

Subjective norms may be measured by asking respondents to think of ‘those important to me,’ or ‘those whose opinion I value,’ or by specifying subgroups based on interviews and focus groups with a representative sample. This is illustrated below:

• Those whose opinion I value think that it is important to update a systematic review every 2 years:

  strongly agree 1 2 3 4 5 6 7 strongly disagree

• My colleagues think that it is important to update a systematic review every 2 years:

  strongly agree 1 2 3 4 5 6 7 strongly disagree

• My review co-authors believe that it is important to update our systematic review every 2 years:

  strongly agree 1 2 3 4 5 6 7 strongly disagree

• My colleagues think that I (use scale below) update my/our systematic review every 2 years:

  should 1 2 3 4 5 6 7 should not

Assessing perceived behavioural control

Assessment of perceived behavioural control should include questions that measure the author' confidence in performing the update and perceived control over performance of the update.

• I am confident in my ability to perform an update in 2 years:

  strongly agree 1 2 3 4 5 6 7 strongly disagree

• Whether or not the review is updated in 2 years is entirely my decision:

  strongly agree 1 2 3 4 5 6 7 strongly disagree

  Compiling a list of conditions that make updating easier or more difficult to perform from a representative sample of reviewers would also be a useful way to develop specific questions to assess perceived behavioural control.

• Conducting literature searches without the assistance of a librarian makes it (use scale below) to complete an update in 2 years:

  much more difficult 1 2 3 4 5 6 7 much easier

Results

Scoring and other measurement issues

Ajzen¹⁵ and Francis et al.¹⁴ provided detailed instructions on designing and scoring each TPB construct. If the triggers to update are time-dependent, it seems important to establish whether the TPB constructs will include a specific time period (e.g. in 2 years) to capture behavioural intent for an upcoming time period or whether the constructs will measure behavioural intent and attitudes towards continual updates (e.g. every 2 years). It also seems important to assess the beliefs of a representative sample of review authors, disperse questions measuring the same construct throughout the questionnaire, and pilot test the questionnaire before mass distribution. Finally, some consideration should be given to the timing and frequency of questionnaire administration as well as method of administration, as intention may vary over time.

Discussion

A theoretical framework such as the TPB not only provides an opportunity to assess authors' perceptions of updating, but also allows review groups to use this information to tailor interventions to motivate authors throughout the systematic review process and stimulate behavioural change where there are difficulties.²¹ The targets of the intervention are perceptions of behavioural control, attitudes and beliefs, and also social norms. Theoretically, changing any of these factors will influence the intention and thus the performance of the behaviour. In addition to maintaining regular contact with review authors in the interim between initial publication of the systematic review and the first, second and third updates, behavioural interventions should target negative perceptions of updating and, when time and limited resources are a concern, include practical services such as librarians to assist with updating searches at regular intervals, assistance with funding for dedicated time to update a review, and ‘update camps’ for authors to work on a review update with methodologists, statisticians, and librarians onsite to provide assistance. These interventions may increase the number of up-to-date systematic reviews and thus provide the best available evidence for use by clinicians, policy makers and consumers.