Clinical academics’ experiences of REF2021

Abstract

The UK Research Excellence Framework (REF) is an assessment of the quality of research carried out in UK Higher Education Institutions (HEIs), performed in 7-year cycles. The outcome impacts the rankings and funding of UK HEIs, which afford the exercise high priority. Much of what REF measures is known to be biased against academics with protected characteristics: for example, women and ethnic minority researchers are less likely to win grants or be published in prestigious journals. Despite changes to REF since 2014, the risk remains that the process might amplify well-recognised existing disparities. The BMA Women in Academic Medicine and Medical Academic Staff Committee carried out a survey of UK clinical academics’ experiences of REF2021. The data indicated the persistence of activities previously characterised as ‘extremely harmful’ in Research England-commissioned work, affecting up to 10% of clinical academics. While acknowledging the limitations of the data, women appeared to be disproportionately affected.

Introduction

The UK Research Excellence Framework (REF) exercise is an assessment of the quality of research carried out in UK Higher Education Institutions (HEIs), performed in 7-year cycles. REF outcomes impact HEI rankings and funding prospects, and good performance in the exercise is afforded very high priority by institutions. In 2022, the results of the latest cycle (REF2021) were published and a consultation on proposed changes to REF2028 recently too place.

Clearly, robust mechanisms must be in place to assess quality of publicly funded research, not least in terms of taxpayer accountability. It is important that these serve to support a healthy, motivating and equitable research environment. In particular, they must not disproportionately impact academics from under-represented groups.

In the 2014 REF, women and certain other groups were under-represented in HEI submissions compared with the pool of all academic research staff.¹ Consequently, the 2015 Stern report² made recommendations to mitigate these disparities, adopted for REF2021. Notable changes included the requirement to submit all eligible academic researchers, in an attempt to prevent institutions ‘gaming’ the system by making selective submissions; and the partial decoupling of staff from outputs, meaning that not every academic had to submit the same number of outputs. This was intended to take pressure off individual academics and to take better account of individual circumstances, such caring for dependants, early career, sickness and disability.

The high-stakes nature of REF means that HEIs have competing pressures and priorities, the power of which should not be underestimated. In REF2021, there were still ways in which HEIs could apply a degree of selection to academics submitted, for example by changing the nature of contracts. Clinical academics employed by universities, but also with NHS responsibilities, are potentially at particular risk from this practice. Their substantive contract in a HEI and their honorary NHS contract means that it is relatively easy to change their status for the purposes of REF. For example, they might come under pressure to reverse the status of their substantive and honorary contracts, or abandon formal academia entirely for the NHS to avoid their inclusion in REF. This comes on top of the wider pressure to move from a research-focused to a teaching-focused role and, thus, no longer be REF returnable; it could also exacerbate the existing under-representation of doctors with protected characteristics in clinical academia.

The BMA Women in Academic Medicine (WAM) and Medical Academic Staff Committee (MASC) carried out a survey in 2022, before the outcomes of REF2021 were published, to explore UK clinical academics’ experiences of REF. Here, we report offer a perspective and experiences of clinical academics on the REF2021 process.

Methods

Microsoft Forms was used as a survey platform. The link to the final online survey was distributed by email to all the clinical academic members of the BMA in April 2022. Some of the data being collected regarding employing institutions could be deemed highly sensitive. Thus, respondents were assured of anonymity and the names of employing institutions were not collected. The survey closing date was before the outcomes of REF were published, to avoid the outcomes influencing individual respondents.

Results

In total, 73 responses were received, with 70 respondees confirming that, during the 2014–21 REF cycle, they were employed by a UK HEI with a substantive (not honorary) contract of at least 0.2 full-time equivalents (FTE), which involved carrying out research, thus meeting criteria for REF eligibility. This is equivalent to 7% of BMA members contacted who are recorded as consultant clinical academics or senior academic GPs employed by universities (the main groups affected by REF). Of the 58 who gave gender and ethnicity, 20 (34%) were female and seven (12%) declared a minority ethnic background. By comparison, 14.5% of the BMA medical academic membership reports having an ethnic minority background, and 40% are recorded as female. Therefore, both groups were slightly under-represented among respondents.

Although the survey cannot claim to be representative, given that those with strong opinions on REF might be more likely to participate, it captured some concerning responses.

Rating of outputs

Academic outputs (usually, in medicine, peer-reviewed publications) are rated during the REF assessment process, from 1* to 4*. Institutions submit those outputs considered to rate highest. REF guidance states that any eligible outputs by an academic can be submitted, regardless of star rating, but our survey found that some institutions stipulated a minimum star rating for submission, meaning that the REF-eligible outputs of some academics were considered ‘ineligible’ by institutions, which might then seek ways to avoid their submission by excluding that academic from the REF process.

The survey asked: Did your medical school stipulate a minimum star rating for papers submitted to REF? Of respondents, 56% answered yes. In most cases (83%), the minimum rating was 3*, although 10% reported a minimum of 2* for submission and 3% reported that a 4* rating was required for inclusion. However. 30% said that they did not know whether their institution stipulated a minimum star rating.

Pressure to change contracts

The survey asked: Were you put under any pressure to change your contract type, job or role e.g. from primarily research to primarily teaching or management, move to an NHS contract, or any other change, in order to avoid having to be submitted for REF? Of respondents, 10% reported pressure to change their contracts because of REF. Notably, all those who came under pressure to change contracts reported having REF-eligible outputs, but half reported that their eligible outputs did not meet the star-rating requirements of the HEI.

Half of the respondents who came under pressure to change their contracts were pressed to move to teaching-focused contracts. Two were pressed to retire or threatened with redundancy. Another stated that they observed other staff being pressed to transfer from research contracts; one stated that they had left their HEI as a result. Half of those who came under pressure to change contracts actually did so.

Comments included:

‘I was pressured to change to a teaching-focused contract, which I did.’

‘It has caused me to change my academic direction to teaching-focused. Before that I was publishing funded research in respectable journals in my field and was quite highly cited. REF process has caused me to cease my research efforts. I have continued to progress my career through teaching and education instead.’

‘pressure to retire or cease role on this round’

‘suggestion to do more teaching and admin’

‘Suggested moving me to a teaching contract at one point - at one point also illegally pressurised towards redundancy’

‘Anyone not meeting the minimum standard for submission was transferred off a research contract - or at least they tried’

‘I left to go to another uni where I was entered’

Comments on why people came under pressure to change contracts included:

‘3* was required. I had a number of publications rated 2* by my HEI’

‘lack of MRC / Wellcome grant income, and probably also publications’

‘research difficult to fit with institutional research strategies’

‘I had zero feedback about the whole process as primary care staff were shunted around’

‘No rational reason’

As well as demonstrating that some HEIs attempted to exclude academics who were eligible for submission according to REF guidance, the comments reflect a general perceived lack of transparency and communication over REF submissions.

REF as a cause of pressure, stress and anxiety

We asked respondents whether they felt under pressure to produce outputs for REF and, if so, the nature and source of the pressure.

Of respondents, 89% reported feeling under pressure to produce outputs for REF, with 38% reporting that this came from more senior staff, who raised the issue of REF outputs directly with them; 26% said it was raised in performance reviews and 15% said it was raised with them in promotion processes. In addition, 55% said inclusion in REF was used as a key performance indicator (KPI) in their HEI; 55% said it was used in promotion processes.

If respondents reported that they felt under pressure from REF, we asked about the impact of that. Responses are recorded in Table 1.

Table 1.

Responses to the question ‘If you felt under pressure from REF, did pressure to produce outputs for REF result in any of the following’.

If you felt under pressure from REF, did pressure to produce outputs for REF result in any of the following:	% ‘Yes’ responses
It distracted me from my teaching work or it caused me to give teaching less attention	26%
It led to a high degree of stress or to mental ill-health	44%
It led it me to work longer hours, impacting on my work-life balance	80%
It led me to fear for my job	50%
It led me to decide to move to the NHS with an honorary University contract	3%
It led me to decide to move to the NHS without an honorary University contract	0
It led me to leave my clinical academic job	0

Comments included:

‘covert bullying’

‘There was a sense that some papers perhaps mattered more than others if they were in particular journals or fitted with others' work. I felt that work on minority subjects I had published on may not have fitted. There seemed a degree of group-think as if we were playing a game of cards and I am not sure that this is what academia should be about. Sometimes it takes years for a piece of work to have real influence’

‘Difficult as now have only 0.2 FTE academic, and work on multiple sites, that is not fully appreciated in producing complex REF outputs, research supervision etc’

‘I had to get the BMA involved to stop bullying behaviours… shaken my confidence in the university and I am considering a switch to just clinical work’

‘I left for another university where my outputs were valued’

‘led me to question whether I was willing to stay in the job. My fear was whether I would want to continue’

‘It wasn't just REF … but the academic environment, which in part, is driven by REF, but I don't think the REF process can be held 100% accountable’

Impact on female academics and academics from ethnic minority backgrounds

Results were analysed by gender and ethnicity. Numbers were low, especially for ethnic minority academics, but nonetheless a few findings stood out.

Gender

Of the 10% of respondents who reported pressure to change contracts because of REF, two-thirds (67%) were women, despite women only accounting for 34% of respondents. In addition, all those who reported being pressured to move to teaching roles were women, and all respondents who said they came under pressure to change contracts despite having eligible outputs, but ones that were not starred highly enough by their HEI, were women. One female respondent changed to an NHS contract with an honorary university contract, instead of a substantive university contract, because of REF pressures; no men reported this. In summary, women were disproportionately affected by both threats of changes to contracts and actual changes to contracts.

Although 90% of women and 89% of men reported feeling under pressure from REF, this pressure appeared to affect men and women differently. Women were more likely to report that they were distracted from clinical work (43% versus 26%), from teaching (36% versus 21%) and much more likely to say that REF led them to fear for their job (64% versus 37%).

REF was raised as an issue over twice as often in women's performance development reviews (PDRs) compared with men's (40% versus 18%), and 2.5 times as often during promotion processes (25% versus 10.5%). Women were twice as likely to answer ‘yes’ to the question: Have you personally experienced any other difficulties regarding REF? (35% versus 16%).

Comments made that related to gender included:

‘When my children were very young, my rate of publication dropped…I was told that unless my publications picked up again…I would be required to explain myself to senior University academic management and that disciplinary action ‘could not be ruled out’…similar review meetings continued to be called at intervals, and feeling under pressure, I opted to change to a teaching-focused academic contract.’

‘Huge gender bias’

‘I was very part time in my early clinical career and had x3 maternity breaks - this really impacts on h index as well as research collaborations…’

‘As a female who has worked part time but who is now full time, the impact of maternity leave, part time working etc is never considered on my overall outputs and career stage’

Ethnicity

Numbers were very low but some findings deserve note. Of the two respondents who were pressed to retire or threatened with redundancy, one (the latter) was a clinical academic from an ethnic minority. The academic who stated that they had left their HEI as a result of REF was also a respondent from an ethnic minority, as was the academic who described suffering ‘covert bulling’ over REF. This was despite the fact that only 12% of respondents were from ethnic minority backgrounds.

Of the respondents, 75% of Asian respondents and 92% of White respondents felt under pressure from REF: however, Asian respondents were more likely to say that this led to a high degree of stress or mental ill-health (67% versus 45%), to work longer hours (100% versus 75%) or to fear for their job (67% versus 48%).

Comments that related to ethnicity included:

‘just getting the feeling I am getting “too old” and they want me out- in particular because I am not British and having foreigners employed is a problem’

Perceived impact of REF on individual respondents’ careers

We asked whether individuals felt that the impact of REF on their careers had been broadly positive or broadly negative (Table 2). Of respondents, 24% felt the impact had been positive overall and 32% felt it had been negative.

Table 2.

Respondents’ perception of the impact of REF on their own career.

Perceived impact of REF on own career	Respondents (%)
Mainly positive	12%
Slightly positive	12%
Neutral	41%
Slightly negative	20%
Mainly negative	12%
Don't know	3%

Positive comments:

‘Positive impact on career; part of REF team overseeing submission and impact statements for our School’

‘Has influenced some decisions about which projects to pursue but they have not been bad decisions’

‘It was a positive, motivating experience, unusually’

‘focused my research effort leading to “better” outputs’

Neutral comment:

‘It demonstrates my success but I think that would be demonstrated just as well without REF using other metrics ie by showing papers, grant income’

Negative comments:

‘It has narrowed the perceptions of what a ‘good academic’ is. And so altered the career pathways, opportunities etc for individuals - and paradoxically altered our capacity to impact. Research isn't valued unless it is 4* on the uni defined REF bucket targets. Work that is potentially changing practice (eg hits KE ‘targets’) may not hit REF targets and so is devalued, not supported…and the people who are doing it the same. We are making ourselves obsolete and lacking in value to the stakeholders we seek to work with’

‘My outputs creatively accounted to bolster weaker members of the department; university restructured to try and look better so ended up in an inappropriate department irrelevant to my research.’

‘I am planning on leaving academia due to the relentless pressure to produce outputs, bring in grant income, supervise PhD students, all while maintaining a busy clinical role.’

Many comments also revolved around the amount of time spent on the REF process:

‘Took up many hours of time that could have been better spent’

‘Lots of work on case studies, which were revised and revised and revised and then rejected. A huge amount of time wasted’

‘Mainly many hours on impact case studies which came to nothing and hundreds of hours assessing papers to see if they were suitable for submission.’

‘The effect of the REF is to take a huge amount of academic time away from research. it produces little of value but wastes taxpayers' resources and time.’

Discussion

Evaluating the impact of research remains a major challenge, which has been approached in different ways.³ For example, the EU's Horizon 2020 programme uses a series of key performance indicators, including metrics such as percentage of highly cited papers, journal impact factor, number of patent applications and numerous others.⁴ Such metrics-focused approaches are very blunt tools with profound disadvantages in terms of researcher diversity and inclusivity.

The data and comments collected here about REF raised concerning themes. Individual accounts from respondents revealed unacceptable behaviours and pressures on some staff, including bullying behaviours (mentioned by two respondents), academics at risk of, and experiencing, changes to contracts (10%), and a high degree of stress (44%) and fear for one's job (50%).

The fact that over half of respondents indicated that their university stipulated a minimum star rating for inclusion of outputs in REF is concerning. Academics should not be excluded from REF because their outputs are not deemed 3* or above, although naturally the HEI will wish to submit the best outputs possible. The guidance is clear that outputs with any star rating may be submitted.

Female academics were disproportionately affected by changes in/threats to changes in contracts. Both women and those from ethnic minorities appeared to be particularly affected by stress and fear for their jobs over REF, with 37% of men, 64% of women and 67% of Asian respondents reporting the latter. One ethnic minority respondent was threatened with redundancy and another left their HEI for another (where they were submitted to REF).

One aim of the Stern Review was to reduce what was seen as unacceptable pressure on grass-roots academics. These data indicate these pressures have not been eradicated, and that activities characterised as ‘extremely harmful’ in the Research England-commissioned report ‘Understanding perceptions of the Research Excellence Framework among UK researchers’⁵ persist, and might affect up to 10% of clinical academics. REF also shapes the type of research that is encouraged. For example, applied research serving NHS employers and the UK health policy agenda might not achieve a 3* ‘internationally excellent’ rating, being deemed ‘UK focused’, with the risk that clinical researchers might avoid this type of work, despite its importance to UK health services and patients.

In mid-2023, following initial review, a consultation on proposed changes to REF28 was carried out.⁶ Initial decisions presented included that staff data would be obtained directly from the Higher Education Statistics Agency (HESA), so that institutions need not need make submissions of individual staff; the link between individual academics and unit submissions would be fully broken; institutions could submit outputs produced by any staff member, including those on teaching contracts; and there would be no minimum or maximum output requirements for individual staff and, therefore, no need for a process to account for individual equality-related circumstances. The consultation invited comment on the practical challenges that institutions might encounter in implementing these changes, how these might be mitigated by funding councils, the impact of these changes on individual researchers (particularly those with protected characteristics), and the potential unintended consequences of allowing submission of outputs produced by those on non-academic or teaching-only contracts.

BMA WAM and MASC welcome this consultation. Previous efforts by REF to introduce protections for staff have been recognised by the research community,⁷ but nonetheless it has remained a high-risk exercise for individual researchers. Much of what is measured by REF is known to be biased against academics with protected characteristics: for example, women and ethnic minority researchers are less likely to win grants⁶ or have their work published in prestigious journals.8, 9, 10 The relative ease with which clinical academics can be excluded from submission by moving contracts from a university to the NHS (with honorary academic sessions) or even losing academic sessions completely, impacts job security for all clinical academics, but those from protected groups might be at most risk. This could amplify well-recognised existing disparities and diminish the appeal of clinical academic careers, to the further detriment of diversity in clinical academia and with potential long-term adverse impacts on UK medical research.

Authors’ contributions

APD initiated and designed the survey and wrote the paper and survey report. All other authors reviewed, advised and commented on the survey draft before it was circulated, and reviewed and commented on the data and survey report.

Declaration of competing interests

There are no conflicts of interest to declare.