Research assessment – Time for a rethink? By Bahram Bekhradnia #REF21

• Following the announcement of the results of REF2021 tomorrow, join us in conversation with David Sweeney, Executive Chair of Research England. Register for the webinar here.

Bahram Bekhradnia has worked in education policy for more than 40 years. Bahram founded HEPI in 2002 and was its Director until January 2014, since when he has been HEPI’s President. Before establishing HEPI, Bahram was the Director of Policy for the Higher Education Funding Council for England (HEFCE), where he presided over several Research Assessment Exercises.

The REF results are out tomorrow. Without knowing the details, it’s possible to make a number of predictions.

1. First, they will show improvement – perhaps significant improvement – in the performance of universities right across the sector.
2. Second, the top 10 performing universities – perhaps the top 20 – will not have changed significantly since the last REF, or indeed since the original RAE back in 1991.
3. Third, there will have been a narrowing of the range of performance.
4. And finally, this will create a headache for those who have to decide on the funding consequences – but one way or another they will see to it that most of those who have improved their performance will receive little by way of financial reward for having done so. It has always been thus since the 1991-92 RAE, and there is no reason to think that the same pattern will not be repeated.

The first full-blown Research Assessment Exercise (RAE) was conducted in 1991-92, following light touch Research Selectivity Exercises in 1986-87 and 1989, which had little by way of funding consequences. The 1991-92 RAE was created at a time when the polytechnics were integrated into the university system, but the amount of money available for research barely increased. And so it was felt that a mechanism had to be found to ensure that the limited amount of money available for research was not spread much more thinly and that the research powerhouses were able to maintain their funding. A credible and rational basis had to be established to achieve this and the RAE was the answer. 

The earlier Research Selectivity Exercise had established the principle of research assessment – motivated largely by the perceived need to satisfy the Treasury that funding provided to universities for research was not simply pouring down what they called a ‘black hole’. The assessment was intended to provide accountability and reassurance about this. The RAE then added the largely new dimension of the funding element, which provided an incentive for universities continually to improve their research performance. It was spectacularly successful in these aims. It ensured that funding continued to be targeted at the highest performing universities while providing a relatively small amount of funding to others in a limited range of subject areas. And it certainly led to an improvement in the country’s research performance.

Lord Bob May, former Government Chief Scientist some years ago provided data that showed the improvement in the UK universities’ research performance compared to their international peers could be directly traced back to the introduction of research assessment – to the point where university research is now one of the few areas where the UK remains world leading. Other countries conduct research assessment and although in some – Hong Kong and New Zealand, for example – funding decisions are based on the results; in others – the Netherlands, for example – they are just part of a holistic assessment of university quality.

So, without doubt, the RAE and the REF have led to improvements in research quality and output, but at a cost. First, the huge financial incentive to put resources and effort into research, and the absence of comparable financial incentives to focus on teaching and learning, is a concern. Indeed, the imperative provided by the REF to focus on research reinforces the fact that, internationally, academic prestige is associated with research. By and large, universities’ reputations and those of academics are not based on their teaching prowess but on their research. So, there is an inherent inclination on the part of academics and academic leaders anyway to focus on research – academics even at less research intensive universities are, after all, members of the same profession as those employed in research powerhouses – and this inclination is reinforced by international league tables which are based almost entirely on one or other measure of research performance. It was this that prompted the policy effort to find a way of incentivising teaching as well – eventually leading first to the National Student Survey and latterly to the Teaching Excellence Framework – though as these have little or no funding consequences the incentives that these provide pale into insignificance compared to the REF.

Second, the REF is a zero-sum game. All universities feel the need to compete and improve their performance – and most will do so – but all this effort is likely to yield scant reward. Everybody improves but the amount of money available does not increase. So, they are running hard to stand still. While universities retain a block grant, and there is no constraint on the way that the money gained from the REF can be used, universities know that if this money is not used to improve their research, their performance will slip at the next assessment, and so their funding will reduce. That was not in our minds as we created the original RAE, but it is a fact contributing to the treadmill that universities find themselves on. They are incentivised to ensure continual research improvement, even at the expense of focusing on other areas of their activity: as the financial rewards are for research, understandably that is where universities feel they must put a disproportionate amount of effort. If they do not, and they slip back compared to other universities, they will suffer financially and reputationally. It is a treadmill they cannot afford to get off.

Finally, and in part because of its very success, the range of performance has narrowed – and so increasing statistical gymnastics are required to enable the REF to perform its initial purpose of differentiating funding and providing significantly more to the highest performing. Over the years, this has led to an increase in the rating scale, adding two points to the five-point rating scale, replacing the seven-point scale with the star system, and adjusting the relative financial values of different points on the quality rating scale. It’s difficult to imagine what further changes will enable funding differentiation to be maintained.

So, the REF has become dysfunctional over time and its days must surely be numbered in its present form. Something has to change – but it is difficult to know what. So long as there is core research funding to be allocated and all universities are eligible to compete for it – and so long as there therefore remains a perceived need to allocate funding differentially – there will continue to be a need for some rational and objective basis for allocating research funding. What is now needed is not continual tinkering with the mechanism of the REF and the subsequent algorithm for converting REF results into funding, but a more fundamental review of the policies that underlie it, the place of research within the university system, how that research is to be funded and the consequences for the academic profession of different decisions that might be taken. And, more fundamentally, whether the disadvantages of the present highly concentrated research funding, and the exclusion of the majority of universities and their academics from significant research funding and activity, outweigh the benefits of having a small number of universities recognised as world leading in research.

This is the fifth in a series of blogs reflecting on the REF. The full list of blogs in the series can be found here.

Register for our webinar with David Sweeney, Executive Chair of Research England, here.