Applying for grant funding is a consuming process that occupies a large part of an academic’s time, as well as their mind and spirit. Many academics will inevitably tell you that acquiring funding is difficult and getting harder due to reductions in research budgets and an increasing number of applicants. In this environment what a researcher wants and needs is a “level playing field” to ensure their grant is assessed fairly and decisions are based on merit and the quality of the science and track record. I have been asked to offer my personal perspective on how grants are reviewed in Australia and in doing so allow you to compare what may or may not happen in India. These views are mine and should not be seen to reflect those of the National Health and Medical Research Council (NHMRC).
This commentary will focus on how we try to ensure fair peer review. It does not deal with what is the best grant system. Therefore, I won’t be discussing issues such as the length of grants, whether we should fund prospective or retrospective research, the number of grant cycles per year, resubmission of grants (i.e.akin tosubmitting a revised manuscript) etc etc.These are major commentariesin themselves and beyond the scope of what I have been asked to write about.
This editorial will draw upon the system used by the NHMRC, which is the major funding body for medical research (both clinical as well as basic science). Since 2005 I have served as a member on 8 NHMRC Grant Review panels, and once as a chairperson. While the forms, rules and computer systems have changed from year to year, the overall principals of grant reviewing have remained fairly constant.
The NMHRC grant review system
Before discussing the pros and cons of the NHMRC system I need to provide a summary of the peer review process so you can understand what happens. I will focus on project grants, which is the main funding scheme that the majority of medical researchers would apply to. But major elements of this peer review process also extend to the other schemes funded by the NHMRC.
Detailed information on the NHMRC funding and review schemes is available online via the NHMRC website: http://www.nhmrc. gov.au/grants-funding/peer-review
What happens to a project grant
-
•
We have one round for project grants each year, with submissions in March and results are announced typically by October-November that year.
-
•
The NHMRC will allocate the grant to a Grant Review Panel (GRP, also called a Peer Review Panel (PRP)) which are discipline based (Neuroscience, Cancer, Cardiovascular, etc)
-
•
A GRP consists of approx. 10–12 members. The members are chosen by the NHMRC administration based on their reputation and expertise for the research discipline of the panel and grant topics submitted that round. A GRP typically contains a mix of basic scientists and clinicians, or from other professions if the panel requires it. There is broad geographical spread and institutional representation, as well as gender balance. There is a mix of senior as well as early career people. Members can sit on GRP for a number of years, with 3 being a typical stint.
-
•
Each GRP will be allocated 80–110 grants.
-
•
GRP members, via an online site, look at all the grants allocated to their panel and have to declare whether they have a conflict of interest with the applicants. They are given the names of the investigators, the institution, project title and project summary. The NHMRC provides a list/examples of what constitutes a conflict and whether it is a major or minor conflict. Some conflicts are automatically deemed major such as co-authorships in the last 5 years, same department or faculty or university a collaborator, or a personal relationship such as a friend.
-
•
Based on the response given by the GRP member the NHMRC administration will decide which grants a GRP member is allowed to assess and which they cannot assess due to a significant conflict of interest. A GRP member can assess grants if they have a minor conflict of interest.
-
•
The grant is then allocated to a Primary spokesperson and a Secondary Spokesperson on the GRP. The primary spokesperson scores the grant and provides written comments and questions. The secondary spokesperson provides scores.
-
•
Each GRP member will be allocated approximately 10 grants as Primary spokesperson and 10 grants as a Secondary spokesperson.
-
•
The grant is also sent to two external assessors who provide written comments.
-
•
The external assessors are selected by another committee, independent of the GRP, called the Assigners Academy. As for the GRP, members of the Assigners Academy are appointed by the NHMRC administration and is composed of individuals with “extensive knowledge of relevant Australian and international health and medical research fields and a reputation for integrity.”
-
•
The Assigners Academy members also have to declare conflicts of interest for the grants allocated to their panel.The Assigners Academy members are allocated 15–25 grants each to find external assessors.
-
•
External assessors are selected and invited by the Assigners Academy member based on their expertise and suitability to assess the topic of the grant.
-
•
Invited external assessors have to declare (and be free) of a significant conflict of interest to be allowed to assess the grant.
-
•
The applicant receives the comments from the Primary Spokesperson and the external assessors and provides a rebuttal to their comments.
-
•
The GRP meets face to face, typically over 4–5 days, to evaluate the grants and score them as a committee.
-
•
At the GRP meeting only members who do not have a conflict of interest participate in the evaluation and scoring of the grant. Conflicted members leave the room.
-
•
The GRP meeting is chaired by an independent Chairperson, either from the NHMRC administration or a senior academic who is typically from a different discipline and with little/no association with the institutes the grants are from.
-
•
The grant is assessed at the GRP meeting as follows: Primary and Secondary spokespersons declare their scores. The Primary spokesperson discusses their assessment and scoring of the grant. Then Secondary spokesperson does the same, and includes the external assessor comments and the rebuttal comments. The grant is then open for general discussion by the other panel members, which can involve questioning the spokespersons about their comments and scores.
-
•
The Primary and Secondary spokespersons affirm their scores – and they can change them based on the discussion.
-
•
All the GRP members present (i.e. not conflicted) vote by writing their scores, in secret, on a score sheet.
-
•
The score sheets are collated and the grant given an overall score.
-
•
If a panel member is voting two or more categories away in either direction from the Primary spokesperson’s scores then they have to openly declare this to the panel, and the reasons and this is noted by the Chairperson and the secretariat.
-
•
Applicants are notified of the outcome.
The tenets for a fair grant review process
Conflicts of interest
It should be clear from the above summary that the NHMRC considers the management of conflicts of interest a priority. This is to reassure their major stakeholders, the applicants and the general public (i.e. the taxpayers who fund the system), that the system allocates government funds fairly and on merit. The obvious concern is that an assessor with a conflict of interest will be more favourable towards a collaborator, university colleague or a friend – whether for personal gain (i.e. funds or resources may come to their group) or to enhance the reputation of their university (i.e. grant success attracts additional government funding and also increases a university’s ranking and reputation). Regardless of whether this does or does not occur, it’s the perception that this could happen which provides sufficient concern and necessitates conflicts to be declared and the conflicted assessors prevented from assessing that specific grant. There are constant reminders to state conflicts throughout the entire review process and if a new conflict arises they must be declared regardless at what stage of the assessment.
Confidentiality
The other important principal of the review system is confidentiality. The GRP members and assessors are bound by a confidentiality agreement that prevents them from discussing the grant and the proceedings of the GRP meeting with a third party. This is to ensure the identity and the comments of the GRP members and the external assessors remains anonymous. It’s a system we are familiar with for refereeing manuscripts. While there has been debate about the merits of confidentiality for manuscript reviewers, I believe the removal of confidentiality in grant reviewing would be problematic. With manuscript submission you have many journals to submit your paper and multiple opportunities during the year. With grant funding the opportunities are very limited and the likelihood of success low. This is not really the case for manuscripts where it’s the author’s decision on how high they wish to aim that can influence if and when the paper gets accepted. However, the consequences of not funding a grant is a lot more profound than not accepting their paper. That decision can be very emotive as it can lead to jobs and labs being terminated due to the cessation of funding. Removing confidentiality would hinder many assessors and GRP members to provide a frank negative assessment and we will have a lot of grants scored incorrectly high.
Transparency
The NHMRC review process is transparent and available on their website. This removes any mysteries associated with the grant review process and applicants can have trust that their grant is handled fairly.The NHMRC also provide detailed guides on the funding rules and instructions to applicants for writing their grants. This is an important component of our system since it helps the applicants understand how their grant is assessed. Not just the logistics I’ve outlined above, but the scoring system. The NHMRC has a set of written descriptors that are used in conjunction with the scores (7 is the highest category to 1 for lowest category) to ensure the score accurately reflects the category the grant is allocated to (i.e.7. Outstanding by International Standards, 6 Excellent, 5 Very Good, 4 Good, 3 Marginal, 2 Unsatisfactory, 1 Poor). Moreover, there is scoring in three categories which carry a different weight for the final score. Scientific Quality 50%, Significance and/or Innovation 25%, Track record of the team 25%. The descriptors are referred to constantly during the GRP meeting by the Chairperson and the other GRP members as a reference point to ensure the scoring accurately reflects the quality of the science and track record. It also allows grants on diverse topics and from different GRPs to be compared.
The other aspect of transparency occurs at the GRP meeting where the GRP members discuss the grant face to face. While still under the protection of confidentiality to the outside world, it does force thespokespersons to justify their views openly and not hide behind anonymity. These discussions are very constructive and robust, and typically done with a strong spirit of collegiality. The Chairperson ensures everyone gets their views heard and they are addressing the descriptors.
Rebuttal
All applicants are given the opportunity to provide a rebuttal to the assessors’ comments. This is an important component of the review system. It allows the applicant to address any misunderstandings by the assessors and to update their track records or announce significant developments with the project since its submission. It is also helpful for the Spokesperson to seek clarification on a matter that if resolved could significantly affect their scoring of the grant. The rebuttal letter is discussed at the GRP, and the Secondary spokesperson has the responsibility to lead discussion on the rebuttal. The panel members will refer to the applicant’s rebuttal letter to see if the major issues have been addressed especially if this is affecting the scoring of the grant. The Chairperson is responsible for ensuring the rebuttal letter is treated fairly.
Oversight and Accountability
Independent observers, called Community Observers, are respected members of the general community who attend the GRP meetings and their role is to report to the NHMRC whether their review process is being followed correctly and consistently. This provides an additional layer of independent surveillance and compliance, and gives the relevant government departments and the public who they are answerable to the confidence that NHMRC policy and procedures are being adhered to as expected.
The Chairpersons meet at the end of each day to discuss the day’s proceedings and the NHMRC administration provide an analysis of the scoring metrics to ensure some panels are not being too generous or too harsh with their scoring. This feedback helps the Chairperson get the right balance in scoring for their GRP as they convey this back to the GRP. Finally, applicants have the right to seek clarification from the NHMRC on the outcome of their application or to make an objection to any part of the peer review process.
No system is perfect
Some grants may, for a number of reasons, not be assessed the same as the others. While this is clearly not ideal, what is important is whether this occurs due to unfair practices or due to inevitable limitations. The emphasis on conflicts of interest can reduce the pool of expert assessors for a grant and the expected level of expert assessment is not achieved. Some research fields are quite small with collaborations amongst the different groups meaning they are all conflicted for their colleagues’ grants. The NHMRC does seek external assessors from overseas in order to increase the available pool of assessors. But it can at times be a difficult process to engage international scientists to support a foreign grant scheme. And some international assessors may not be familiar with the language and scoring of the NHMRC system and they provide comments or scores that are not congruent with the local assessments. Conflicts of interest can also have a significant impact at the GRP meeting with expert members having to leave the room due to their conflicts and the remaining members do not have the necessary expertise to properly assess the merits of the science. This can work either for or against the grant since critical flaws in the science could have been missed or the strengths of the project were not properly appreciated. This could result in the grant getting an undeserving higher or lower score compared to the other grants on the panel which had the necessary expertise in the room. This is a difficult scenario to manage whilst also trying to ensure conflicts are correctly implemented. However, if there is a real concern then the Chairperson can ask a conflicted member to comment on just a particular aspect of the grant to provide clarification for the GRP (but not score the grant), or seek a member from another panel to step in for their comment. In my experience the Primary and Secondary spokesperson work hard to ensure they have read the grant carefully and sought clarification on matters they were not experts in and in most instances the panel seeks to assess the grants properly.
Another issue with conflicts of interest is they are mainly declared for those who are conflicted in a “positive” way. The system does not really allow those who have a “negative” conflict to be readily identified. This would include competitors who could mark a grant low to hinder the applicant’s progress, or someone who has a very narrowminded or biased view of the science and is very negative. It has also been suggested that a potential flaw in eliminating those who are from the same institute as the applicants is that you are only left with GRP members and assessors from rival institutes, and they could be un-supportive of grants and score them down in an attempt to make their grants or those from their institute finish higher up the funding list. While human nature is varied (in good and bad ways) and you can never be sure of a colleague’s motives, it is unlikely this latter scenario would be prevalent or successful. It would require collusion by a number of members with a similar goal.Moreover, the panels are composed of members from a number of institutes, and there is the Chairperson and the other members challenging their views, especially if the science is very solid. However, since the scoring is anonymous it could lend itself to be manipulated in this way.I think the risk is very low, and more of a conspiracy theory, and the need to implement conflicts of interests should take precedent.
Summary of what makes a good grant review system
-
1.
Clear guidelines explaining the review process and the scoring system.
-
2.
All guidelines and information documents to be made available.
-
3.
Adherence to the guidelines and scoring system by the funding organization and grant review committee.
-
4.
Declaration of significant conflicts of interest.
-
5.
The grant review panel is composed of members with diverse skills, with expertise for the grants under review, are for different institutions and the members are both senior as well as junior-mid-career.
-
6.
Regular turnover of the committee membership to open the process to more people and avoid cliques forming.
-
7.
Face to face meeting of the grant review committee for the final assessment of the grants.
-
8.
Independent and trustworthy Chairperson to mange the grant review committee meeting. Preferably with no significant allegiances/associations to the field of research and institutions seeking funding.
-
9.
Opportunity for the applicant to submit a rebuttal.
-
10.
All aspects of the process are audited by independent personnel and declared publicly.