In response to complaints about reforms to peer review of funding applications, the Canadian Institutes of Health Research (CIHR) has announced modest modifications. But numerous senior scientists, including two leading researchers recruited by CIHR to help guide its reforms, say the agency will have to do far more to re-establish their faith in its all-important peer-review methodologies.
At issue are reforms introduced in 2014 that reduced CIHR’s reliance on face-to-face peer review panels, while moving much of its application and scientific review functions online. The face-to-face panels typically involved CIHR convening groups of between 8 and 20 leading subject experts in Ottawa to confidentially assess applications for funding
The new “remote virtual screening process,” CIHR explained when it announced the reforms in 2012, brings combinations of peer reviewers together in a “virtual space” to assess funding applications in order “to gain cost-effective access to a broader base of expertise (including international experts), reduce biases that occur in face-to-face discussions, and reduce the burden of travel demands imposed on peer reviewers’ time.”
Laudable as these aims are at a time when the CIHR is under substantial budget pressure, Jim Woodgett, director of Research at the Lunenfeld-Tanenbaum Research Institute in Toronto, says that in “trying to create a peer review machine for fairness,” the CIHR risks severely damaging the quality of its peer reviews.
“The [peer-review] panels have the virtue of people having to defend their scores [for funding applications], which is now largely gone when you have a virtual process,” says Woodgett, who has frequently chaired CIHR’s cancer research review panels. “There is a severe loss of interactivity between scientists.”
The new virtual peer review to determine who gets CIHR research funding is garnering complaints.
Image courtesy of suphakit73/iStock/Thinkstock
University of Alberta cell biologist Richard Wozniak is worried that the new process might adversely affect the quality of scientific decision-making. “I am concerned my science might not be appropriately reviewed.”
The crux of the issue, argues Lunenfeld-Tanenbaum Research Institute cancer investigator Rod Bremner, is that many of the free-flowing, confidential discussions that often occur among review panellists behind closed doors likely cannot be replicated online. “Trying to do these discussions online is a nightmare. There’s a reluctance to type certain things. When you are together in a room, you are much more committed to the process than when it’s online.”
A Dec. 12, 2014, letter signed by Bremner and six other scientists warned CIHR that the new system “severely limits the use of reviewers with specialized knowledge.”
This echoes an earlier letter signed by 50 scientists at the Ontario Cancer Institute in March, 2012, warning CIHR President Alain Beaudet that “the proposed changes in the review process are seriously flawed” and “there is particular concern about the use of anonymous, on line reviews.”
With this in mind, CIHR will continue to convene face-to-face panels for some of the toughest-to-decide “grey area” cases, says Jennifer O’Donoughue, CHIR’s executive director for Reforms Implementation. “We were told the old system was excellent,” she acknowledges. “The panels are not being totally abolished.”
As the reforms are implemented, O’Donoughue adds, the CIHR is closely monitoring reactions. Preliminary data from a survey completed by 70% of 443 reviewers who participated in CIHR’s first-ever virtual review exercise show “trends in a positive direction,” she says. The reviewers who completed the survey were among those who assessed 1366 applications within Phase 1 of the CIHR’s first-ever Foundation Scheme awards, which draws on a $500-million funding pool.
The survey found that 55% of respondents felt CIHR’s new system “of structured reviews” reduces workload from the previous system of “unstructured reviews.” The survey also found that reviewers generally endorsed the utility of online discussions.
Less encouraging, however, was the finding that 60% of respondents said CIHR’s new peer review system is inferior to the old system in giving feedback to applicants. And only 55% said they were confident that “their online discussion was considered by others”, suggesting that as many as 45% doubted the validity of the online discussions CIHR now relies on.
Speaking at a Jan. 16, 2015, Town Hall presentation in Toronto, CIHR’s Chief Scientific Officer, Dr. Jane Aubin, promised a set of modest modifications based on the survey feedback, including establishing a virtual chair role to shepherd sets of applications “and ensure that online discussions are being held for applications with discrepant reviews.”
Aubin also pledged to provide more comprehensive reviewer training materials, and a “new rating scale with more gradation at the higher levels.”
This article is part of an ongoing series about reforms at CIHR. Previous articles include: “CIHR excludes Aboriginal health in review” and “CIHR reforms contradict consultant reports.”