Tales from REF Central: Reflections from REF results week

This blog was written by Dr Elizabeth Gadd, Research Policy Manager at Loughborough University.

I’ve spent my whole career in higher education, but this has been my first REF at the coalface. And so this month was my first in ‘REF Central’, our institutional hub of spreadsheet-wrangling and story-spinning, fuelled by a bottomless teapot and an excessive amount of cake. I’m not short of an opinion on the REF: its shortcomings and its possibilities. I even had a thing or two to say about REF results reporting. But nothing really prepared me for the rollercoaster of REF results day.

1. Every number (doesn’t) tell(s) a story

As your results data starts coming through, first in raw spreadsheet form, then processed by your institutional planners, your eyes are hungrily darting across the numbers looking for the big ones. What percentage 4* did we get? Overall? In outputs? Impact? Environment? And how does that compare with the sector? 

And then you might start generating indicators: Grade Point Averages or Research Power ratings (GPA x FTE of submitted staff). You may even start throwing the numbers at a modeller to try and predict what this will mean financially – or at least what percentage of the overall 4* share you had relative to last time. 

But the more you churn the data, with every layer of aggregation and abstraction, the further you get from the humans beneath.

I found their faces appearing ghost-like in the spreadsheet cells. The new Head of Department who felt the weight of the REF so keenly: I feared it might break him. The Unit Lead who had to devise a robust way to fairly select 500 outputs from 5000. And the Associate Dean who spent the last few days prior to submission frantically seeking publisher confirmation that an output would have a 2020 publication date. 

Beyond our own staff lay all our impact case study beneficiaries: the lives immeasurably improved by research-informed social policies and technologies I could only Google.

And, of course, this was largely all delivered in pandemic-struck conditions to a backdrop of dog-barking, doorbells and Dora the Explorer. The laughter, the anger, the anxiety, the hope. The people. All reduced to hundreds of datapoints swimming on a page that frankly by the end of Day 2 I never wanted to see again. Unlike the people. Who I’d gladly do it all again for. And that’s saying something.

I’ve reflected before on the irony of the REF priding itself on being a peer-review process that ultimately results in a spreadsheet full of numbers. But the sheer reductiveness of the the REF has never struck me so hard or left such a bitter taste in my mouth. ‘I hate the REF’, I said to my Pro Vice-Chancellor of Research as he strolled around REF Central. He thought I was joking.  

2. A healthy sector has a range of results

Of course one of the problems with digging only for the biggest bones in the REF results data mine is that we end up ignoring all the smaller ones. But (with apologies for the shaky archaeological metaphor), it’s in their combination that the full treasure is found.

With the change in REF rules this time around (decoupling individuals from outputs so you could submit your best n outputs overall) an increase in 4* volume was inevitable. This didn’t stop the joint funding bodies celebrating, without irony, the increase in 4*-rated submissions as an indicator of a healthy sector. 

But just as a healthy institution hosts those at the beginning of their career as well as those at the end, so a healthy sector also has a range of institutions across the research intensity spectrum. Similarly, some institutions branch out into new discipline areas where they are not yet established. Some disciplines take portfolio approaches to their outputs with nascent ideas (two-star) discussed in chapters and articles that generate discussion which ultimately leads to a life’s (four-star) work. This is normal. 

In our Day 1 REF comms meeting I found myself arguing for the value of a two-star output. It’s internationally recognised, I cried, this is a wonderful thing! Indeed, this is something REF Panel C Chair Jane Millarreiterated in Tuesday’s Institutional REF Debrief, stressing their value in underpinning impact. (Impact case studies have to be based on at least two-star outputs). 

The problem with the competitive culture that the REF engenders is that those with inter/nationally recognised research are seen as losers rather than beginners, or portfolio-builders. This view is reinforced when, for ease of viewing, REF data are RAG-rated (red-amber-green) to highlight our various ‘strengths’. To my mind this is literally de-grading.  I refused to RAG-rate my spreadsheets, choosing instead shades of green. The Vice Chancellor gave me a quizzical smile. There’s nothing poor in our submission, I clarified. It’s all good, and some is very good.  

3. The right of reply

Frankly speaking, I don’t believe all our results. That’s not sour grapes. Or me thinking that poor Professor X deserved better. I just think in some units the scores just don’t smell right. And given what we know about the assessment process, that it was very few people (100 per cent human), assessing a very high volume of submissions in very little time, that’s entirely possible, right?

We also seem to have one mysteriously unclassified output. And I know it’s noise in the light of the overall result, but I still want to know what that was and why it was deemed unclassified. Even some indication as to whether it was an Open Access issue (shouldn’t have been) a date mishap (it fell outside the time-window) or a grading matter (it’s not even nationally recognised) would at least give us a steer. But unfortunately, with REF results there are no queries and no appeals. And unless the panel chooses to make reference to it in their written feedback (due in June) we’ll just never know. 

Given the REF made sure our colleagues had the right to appeal our REF-related decision-making, and that other higher education institutions had the right to complain that we didn’t stick to our Codes of Practice, it feels a bit rich that we have no right of reply at all. I know we’d be here forever if everyone could appeal everything, but maybe giving each institution three disclosures and one appeal would go some way to demystifying some of our odder results?

4. Unpaid and emotional labour

Finally, at the end of Day 2 when all our results were known, we held a private celebration with all those who’d supported the REF most closely. Our Pro Vice-Chancellor for Research rightly congratulated everyone on their efforts, including highlighting the invisible work that goes on in support of a REF submission. ‘Unrecognised and unseen’, he called it. ‘And unpaid’, I muttered.

Because that is the reality of so much of what we do in academia, and research managers are no exception. Whilst our academic colleagues worked into the night writing and rewriting institutional environment statements until they ‘answered the exam question’ within the word count, those of us in REF Central were doing equally burdensome but significantly less meaningful tasks. Mine included sobbing in the toilets as I read through 130 individual staff circumstances submissions. And then sobbing again as I realised that their agonising combination of personal tragedy, physical and mental health issues only entitled their Unit to a reduction of half a fricking output. Yes, I’m still angry.

One of my colleagues was regularly up until one o’clock in the morning in the weeks before the submission, technical checking impact case studies to ensure the rules were minutely followed, that fonts were the right size and margins didn’t run over the page. This is the reality of a REF submission. Not glamorous, not strategic, and not always paid.

I know the joint funding bodies are now gearing up to assess the cost of REF2021. I hope they factor in the free labour and the emotional costs. When we talk about reducing the burden, we don’t just mean the volume of work. 

Summary

The night before the REF results came out, I found myself musing on the curious breed of research manager who does the REF more than once. Unless the approach for REF2027 changes significantly, I hope I’m not one of them. 

This post is part of a series of blogs reflecting on the REF. The full list of blogs in the series can be found here.