This blog was kindly contributed by Dennis Sherwood, who has been tracking the goings on at Ofqual in relation to this year’s public exam results for HEPI.
Whenever I hear that cliché, I shudder: its purpose – especially when used by a politician – is to deflect attention from a previous woeful decision, to avoid the blame.
Hindsight might indeed be wonderful. But it’s foresight that’s important.
For foresight is demonstrated when a decision has to be taken, when a choice has to be made. Choices have consequences, and it is the job of the decision-maker to make the very best choice possible. And that’s the choice that passes the most stringent test there is, the test of time. When we look back, we must all agree, ‘Yes, given the information then available, that was a good decision’. That is the only way we can judge the wisdom – or otherwise – of decision-makers. And, if necessary, hold them to account.
As I write this, the IB [International Baccalaurate] is in turmoil, and – following the submission by schools of their ‘centre assessment grades’ and the corresponding student rank orders – Ofqual and the exam boards are in the process of using their ‘statistical standardisation’ algorithm to ‘make sure that grades are fair between schools and colleges’.
The details of precisely how this algorithm works have remained obscure, and so, in a recent report, the Education Select Committee recommended that:
Ofqual must be completely transparent about its standardisation model and publish the model immediately to allow time for scrutiny. In addition, Ofqual must publish an explanatory memorandum on decisions and assumptions made during the model’s development. This should include clearly setting out how it has ensured fairness for schools without 3 years of historic data, and for settings with small, variable cohorts.
Ofqual, however, subsequently refused, but in a recent symposium, a perhaps rather dim light has been thrown into this murky corner. The slide pack from this symposium has been published, and, to me, the slides contain some good news (small cohorts are explicitly recognised as requiring special treatment), some bad news (the appeals process is still very technical and narrow), and the sad, but perhaps unsurprising, news that the ‘”vast majority” of schools gave “optimistic” GCSE and A-level grades that would have meant unprecedented rise in results’.
It now seems that many centre assessment grades will be binned. Given that the algorithm only needs schools’ rank orders, why were they required at all? That’s just one question about this year’s process…
Two alternative histories
So here are two alternative histories about how grading GSCEs and A levels might have been, had Ofqual taken a different decision after Gavin Williamson’s statement of 20 March.
I believe the two choices I’m about to describe were open to Ofqual at that time, and draw only on information then available. So this is not about hindsight. It’s about foresight.
1. ‘No grade inflation’
Ofqual explicitly declares ‘no grade inflation’ to be the overarching policy, as achieved by constraining each school to its historical grade pattern. The exam boards know this pattern for every subject and every school; they also know how many candidates each school has entered for the 2020 exams. The exam boards can therefore use whatever algorithm they wish to calculate how many grades each school is ‘allowed’ for each subject (which could well be what the current standardisation algorithm is actually doing anyway).
The boards then send a form to each school saying ‘For this year’s  candidates in [GCSE Geography], your school is allowed [so many] 9s, [this number of] 8s… Please enter the names of the candidates to be awarded each grade.’
Schools are not being asked for centre assessment grades or rank orders. All they have to do is put the right number of names in the right boxes, paying particular attention to which side of each grade boundary any name is placed. And surely focusing on the grade boundaries, and being as fair as humanly possible across them, is by far the most important aspect of all this to think – and indeed worry – about.
A board might also allow a school to exceed a grade allocation, provided there is robust evidence. This puts ‘no grade inflation’ at risk, so it all depends on how much ‘wriggle room’ Ofqual might allow.
2. ‘Trust the teachers’
Ofqual says ‘we trust’, and asks for grades based on teacher judgement. Since everyone understands that awarding a million A*s and five million 9s is daft, Ofqual also says, ‘please avoid grade inflation, and behave with integrity’.
Ofqual sends every school a spreadsheet, or provides an online facility, to calculate grades based on history, and dealing with arithmetical details such as averaging, rounding, and year-on-year variability. Since all schools do the computations in exactly the same way, making exactly the same assumptions, the numerical playing field is truly level. The results of the spreadsheet can then be adjusted, as each school wishes, to recognise individuals and also contexts that do not conform to historical averaging.
The validity of these ‘adjustments’ depends critically on the integrity of the teachers who make them. Accordingly, neighbouring schools mutually agree to act as external examiners, so helping temper over-optimism and eliminate bias. And with their wider perspectives, bodies such as ASCL, NAHT, HMC, the Sixth Form Colleges Association and the unions help organise the ‘mutuals’, review pre-submissions, ensure consistency of standards, test outliers, and, most importantly, suppress ‘gaming’. By working together in this way, a ‘Tragedy of the Commons’ is averted, resulting in a powerful demonstration that self-regulation works, and that teachers can indeed be trusted.
The schools then submit their grades, which the boards check, going back to the schools for explanations of outliers. Since these have been pre-vetted, there are not that many, and those that are present are well-evidenced.
On aggregating the results, Ofqual expects, and allows, some modest grade inflation. Adherence to an arbitrary statistical rule does not sacrifice fairness to each individual candidate.
The fair appeals safety net
Both alternatives have an appeals process like Scotland’s: free, and based on ‘further, evidence-based consideration of grades if schools and colleges do not think awarded grades fairly reflect learner performance’, with students being able to seek support from their schools if they feel they have been awarded an unfair grade.
Water under the bridge?
Those are just two possible alternative histories. There may be others. May I invite the imaginative to post comments accordingly!
And as I’ve said, all this was, I believe, totally foreseeable on March 20; nor is anything ‘rocket science’. Were these possibilities examined, and rejected? If so, given what has so far happened, was that rejection wise?
I don’t know how the choice to follow the process that is currently taking place was taken. I wasn’t around the table. And of course my descriptions are superficial, lacking detail on important matters such precisely what evidence is needed to support an outlier; how ‘moderation mutuals’ might be set up; how, in practice, a school might be influenced along the lines of ‘those grades look rather high…’; how boards might most effectively engage in a dialogue with schools to enable them to explain their outliers. And many other things too. But these could all be addressed and resolved, and a practical procedure developed. Over the nine weeks between the original announcement of the cancellation (18 March) and the publication of Ofqual’s consultation decisions (22 May), there was plenty time to get things right.
All that, of course, is water-under-the-bridge. Yes, I can hear the chorus of ‘It’s all very well for this smart Alec to be wise after the event! And who is he anyway?’ Indeed. So don’t rely on what I’ve written. Think about it for yourself.
But it isn’t quite all water-under-the-bridge yet.
To me, a very important and as yet unresolved issue concerns appeals. After much public pressure, the IB has announced it will ‘work with schools to review “extraordinary cases” for appeals’.
The author gratefully acknowledges his conversations with Huy Duong, Mike Larkin, and – especially in connection with ‘no grade inflation’ – Rob Cuthbert.