The flurry of important new documents from the Office for Students will be pored over for months, particularly between now and the end of the consultation period in March 2022.
Most notably, the OfS wants to assess institutions on their students’ continuation, completion and progression rates. Institutions that fall below certain thresholds could be subject to a range of (very) heavy punishments.
The Minister for Universities, Michelle Donelan, says:
Students will be able to select their course knowing that, like the food in their fridge or the car on their driveway, their course has reached a minimum acceptable standard for quality and outcome.
The proposals are radical and unprecedented in approach, scale and detail. But they come with risks. And while there’s lots to say on the minutae, of course, policymaking is also about the big picture. So we must avoid the trap of failing to see the wood for the trees.
Many people will readily agree with Ministers and regulators that it is appropriate to assess courses in a more granular way than in the past. It certainly feels reasonable for policymakers to question the value of their investment in education, even though it is a remarkably hard question to answer in practice.
Those who think differently, who oppose return-on-investment numbers for subjecting the education system to a dessicated calculating machine, are never able to provide an adequate answer to one vital conundrum. How do you convince the Treasury of the value of education if the education system is less willing than other areas of public spending to undertake proper cost-benefit analyses? After all, every £1 spent on education could be spent on something else.
Anyway, it is clear which way the wind is blowing. That’s why, just a couple of days before the OfS documents landed, Universities UK finally published their long-awaited ‘new framework to assess the value of university courses to students and wider society.’ The direction of the wind also explains why the response of the mission groups to the OfS’s proposals are characterised by equanimity.
Super Crunchers
As I read the voluminous new documentation from the OfS, I thought of nothing so much as the persuasive 2007 book Super Crunchers: How Anything Can Be Predicted by Ian Ayres, which argues that data is a better guide than expertise.
As the review of the book in the New York Times noted, Super Crunchers starts with an anecdote on how the world’s best wine-tasters were bested by hard meteorological data:
What is it about math that can make people so angry? In the late 1980s, a wine lover named Orley Ashenfelter began publishing a newsletter called Liquid Assets that predicted how good each Bordeaux vintage would turn out to be. Instead of basing his judgments on the taste or smell of the wine in its early stages, Ashenfelter, an economist at Princeton, preferred data. He had come to believe that the weather during a growing season in Bordeaux was a remarkably accurate predictor of the eventual price of the wine. A hot, dry year seemed to make for great Bordeaux.
As you might expect, wine critics didn’t take very kindly to the professor’s ideas. A British wine magazine denounced their “self-evident silliness.” Robert Parker called Ashenfelter “an absolute total sham.” Eventually, Wine Spectator stopped accepting advertisements for newsletters, apparently because of Liquid Assets.
But even if the super crunchers are right that numbers can tell us most of what we need to know on specific issues, it remains the case that in education almost no one accepts the argument that metrics are everything.
- No parent would be happy if all they received about their children’s schooling were some numbers.
- Teachers would not respect any full Ofsted review if it just involved looking at the numbers for their school.
- Students in higher education get rightly frustrated if their work is returned with a grade and no comments.
The problems with metrics
I read Super Crunchers just as I first started work in higher education policy. At the time, I briefly wondered if it provided a post hoc argument for Gordon Brown’s proposals to introduce a new metrics-based way of evaluating research in place of the old Research Assessment Exercise (RAE).
But Brown had quickly performed a u-turn (based partly on HEPI’s output at the time) and it’s worth recalling why. Technopolis recounted the story in their 2018 report on research assessment, which is worth quoting at length:
Although the RAE was judged as cost-effective, there were still questions about whether the peer review based system was an effective system for the allocation of resources. In fact, the Chancellor of the Exchequer (Gordon Brown) had refuelled this debate already in 2006. In spite of the report by Roberts (2003) and the analysis by Wooding & Grant (2003), the Chancellor of the Exchequer suggested that the next exercise should be replaced with a metrics approach to QR [Quality Related] research funding – if the majority of HEIs [higher education institutions] were in agreement. The government report ‘Next Steps’ (HM Treasury, 2006) presented evidence that the correlation between an HEI’s QR and Research Council income streams, when measured at an institutional level, is as high as 0.98 on average, suggesting that if QR funding were allocated with reference to research council funding there would be little change to institutions’ income. The RAE was seen as unjustified in light of this idea, but also because it imposed a burden on HEIs, influenced behaviour in publishing and recruitment and failed to support interdisciplinary research and excellent user-driven research (HM Treasury, 2006).
In response to the Chancellor of Exchequer, Bekhradnia (2006) published a short evaluation of alternatives to the Research Assessment Exercise and showed that because of differences in size, the percentage change in QR received would be substantial if the allocation of QR funding were based on income derived from research grants and contracts. Moreover, the report by HEPI concludes that a metric based system would cost ‘very much more’ and would likewise influence behaviour (see also the study by Sastry and Bekhradnia (2006)). In his report, Bekhradnia also lists additional downsides to a metrics system. A related study by HEPI also concluded, amongst other things, that a citation-based approach does not measure research quality (Bekhradnia, 2006).
Shortly after, the Department for Education (2006) published its consultation results on the design of a new metric system in which five models for allocating funding on the basis of income and research volume were presented. The consultation produced a range of arguments for and against a metrics based system (see also Barker (2007) for a summary). This consultation highlighted opposition to the reform, confusion around the design options, and concluded that the next exercise would (only) “use metrics as its basis where they are sufficiently robust and fit for purpose, and would help to stimulate development of metrics fit for the purpose of assessment in areas where they are currently less well developed” (pt. 86).
These problems summarised by Technopolis explain why, as I read the OfS papers, I felt more sympathy than before for those who find joy in complaining about the so-called ‘metricisation’ of higher education.
There is a common, even clichéd, argument that higher fees, the removal of student number caps, reliance on international students and the other things said to make up a ‘neoliberal marketisation’ project have put an excessive focus on the numbers and too little focus on all the other hallmarks of a good education.
Yet the neoliberal project, if it even exists, is years – perhaps decades – old and never before has the focus on the numbers been as great as it is now to be. Indeed, increasingly, there seems to be an inverse relationship between the amount of regulation and the level of public funding. Less funding has meant closer oversight.
To be fair to the OfS, they doubtless felt the need to shift towards a more numbers-based approach not only because of pressure from Ministers but also after losing a court case on the basis of their old approach to assessing quality.
Moreover, the OfS have rightly worked hard to introduce more contextual information into their proposals than previously expected:
we recognise that using our indicators to measure the outcomes a provider delivers for its students cannot reflect all aspects of the provider’s context (as some contextual factors are not always measurable through the data we hold). If a provider delivers outcomes for its students that are below a numerical threshold, we will make decisions about whether those outcomes are justified by looking at its context. This approach would result in a rounded judgement about a provider’s performance. This being the case, a provider’s position in relation to a numerical threshold would not, by itself, determine whether a provider was compliant with our condition.
Nonetheless, on reading the new paperwork, it is hard to escape the conclusion that the new measures are not only a very bureaucratic answer to the question of value but may also still put excessive weight on the numbers rather than other evidence.
Holding back The Metric Tide?
Here, it is worth revisiting the 2015 report The Metric Tide, which looked at the use of metrics in our sector (principally in relation to research).
Tellingly and sadly, the link to the report on the Office for Students website no longer works (perhaps because the OfS has no direct responsibility for research), so it is not easy for people to access the whole report. But the Chair of the group that produced the document (James WIlsdon) summarised it at the time as providing the case for a tripartite ‘variable geometry of expert judgement, quantitative and qualitative indicators.’
He went on to say:
Metrics should support, not supplant, expert judgement. … we found that peer review, despite its flaws and limitations, continues to command widespread support. We all know that peer review isn’t perfect, but it is still the least worst form of academic governance we have.
As the sector gears up to respond to the OfS, the first key question to ask is surely: have we got the overall balance between the quantitative data and people’s judgements right?
As part of this, it is only fair to note that the new measures are only part of the how the OfS regulates institutions, with – for example – simultaneous new expectations on access.
Like others, I worry the pendulum may have swung too far in one direction. As a Tweet from the journalist John Morgan yesterday put it:
How complex, potentially, is the system of absolute numerical baselines on ‘quality’ that the OfS is proposing to use to regulate universities and colleges? Well, it took them 640 pages to explain the proposals.
Secondly, if it is all too imbalanced towards the metrics, can anything be done about it? I retain enough faith in evidence-based lobbying to think marshalling enough hard evidence could push the OfS back a little more towards a broader assessment of higher education.
But in the context of ‘tides’, it is also worth recalling the story of King Canute. Today, people like to portray him as naïvely thinking he had the power to stop the incoming tide.
In fact, he was trying to show people that he did not have that power. And so his feet got wet.