This blog was kindly contributed by Dennis Sherwood and Rob Cuthbert. Dennis runs the Silver Bullet Consultancy and is a frequent contributor to the HEPI blog. Rob Cuthbert is an independent academic consultant and Emeritus Professor of higher education management. @RobCuthbert
On the 11th July, The Education Select Committee, chaired by Robert Halfon, published a report entitled, “Getting the grades they’ve earned – Covid-19: the cancellation of exams and ‘calculated grades’ ”. It is a compelling read.
Recognising the urgency and significance of a number of issues associated with this year’s grades, the Committee has published its first report even while it is still taking evidence on other matters. If effectively implemented, the Committee’s recommendations will ensure that more students are awarded a fair grade than would have happened otherwise. That said, there are still some further problems to address.
For this year’s GCSE, AS and A-levels, schools were asked in June to submit, for each subject, proposed ‘centre assessment grades’ for each student, as well as a top-to-bottom rank order. Since then, the exam boards have been carrying out ‘statistical standardisation’ to ‘make sure that grades are fair between schools and colleges’, and ‘that at a national level, grade distributions are broadly in line with other years’ .
If the results of the standardisation algorithm do not match a schools’ submitted grades, the exam board, unilaterally and without consultation, will over-rule the school’s recommendations, and ‘slice’ the school’s rank order as they wish to determine the grades to be awarded in August.
For this process to be trusted, three conditions must be fulfilled:
- Teachers must submit unbiased information.
- Statistical standardisation must be seen to be fair, reliable and trustworthy, so that every individual student is confident of being awarded the grade he or she truly merits.
- There must be an easily accessible appeals process, providing a safety net for students with robust evidence supporting their belief that their awarded grade is lower than they merit.
Should any of these conditions not be fulfilled, there will be trouble, as the International Baccalaureate has already shown – to quote a recent headline “IB results ‘scandal’: Why 2020 grades have sparked fury”.
The most significant recommendations relate to statistical standardisation, for it is this process that determines candidates’ grades. But no one knows how; and there have been fears that the algorithm might unfairly disadvantage, for example, schools with small cohorts, schools on a steady improvement path, and bright students in schools with relative weak past performance:
Ofqual’s decision not to include trajectory in their standardisation process was criticised in the Sutton Trust’s written evidence, which suggested that the ‘turnaround’ schools disadvantaged by this decision “are likely to disproportionately serve poorer communities”.
A submission from University College London’s Centre for Education Policy and Equalising Opportunities warned that the use of historic performance data for standardisation could penalise “atypical” students such as high achievers in historically low-performing schools.
Ofqual’s failure to publish the details has been strongly criticised, for example:
However, little detail has yet been published on Ofqual’s model, and we agree with the Royal Statistical Society’s conclusion that “more transparency” is needed urgently.
Having considered these, and many other, written submissions, as well as taking oral evidence, one of the Committee’s key recommendations is that:
Ofqual must be completely transparent about its standardisation model and publish the model immediately to allow time for scrutiny. In addition, Ofqual must publish an explanatory memorandum on decisions and assumptions made during the model’s development. This should include clearly setting out how it has ensured fairness for schools without 3 years of historic data, and for settings with small, variable cohorts.
This specifies two requirements: Ofqual must publish the details of statistical standardisation ‘immediately’, and also ensure fairness, a concept of many dimensions.
Fairness between one year and the next is supposedly addressed by the standardisation model, which fixes the national distribution of grades. But that kind of collective fairness not only denies the possibility of generational improvement, it may also still allow collective unfairness to particular groups, especially the disadvantaged and some minority ethnic groups. And it may involve massive individual unfairnesses as the model downgrades many students without consultation. In France, for example, the government policy in the exceptional circumstances of this year has been to avoid penalising students taking the French Baccalaureate, allowing some grade inflation rather than risking many students getting grades lower than they merit.
Fairness may also involve issues of bias or discrimination, and the report pays particular attention to the possibility that BAME students, those with special educational needs, and those on free school meals might be treated unfairly.
Since the algorithm has so far remained hidden, we cannot know how or whether any bias is embedded either in the algorithm or the way it uses historic data. It is also possible that there is bias within this year’s centre assessment grades, especially given the known bias in predicted grades for university entrance. This, however, is a distraction. The context for this year’s centre assessment grades is very different from university entrance, as has been made clear in Ofqual’s Guidance Notes; more importantly, centre assessment grades are surely irrelevant in that some can, and will, be over-ruled by statistical standardisation if an exam board so chooses. Indeed, we wonder why they were asked for in the first place – all the exam boards need is the student rank order.
The rank orders – which the exam boards have committed not to change – are fundamental to this year’s process. If a teacher, deliberately or unconsciously, has been biased, the place to look is in the rank order, which is nothing to do with the boards or with Ofqual, but the school’s responsibility, alone. The quest for bias might therefore end up quite close to home, which might be uncomfortable, but is probably not surprising.
So what can be done by students who believe they have been treated unfairly?
One recourse is to the autumn exams, but this option is fraught with problems.
In the understandable rush to introduce a completely new system, after the Secretary of State’s announcement on 20 March, it probably seemed reasonable at first to invent a system in which dissatisfaction could be tackled by an opportunity to take an autumn examination. Over time this choice has unravelled. If initial results match the allowed national distribution and autumn exam candidates succeed in achieving higher grades, then grade inflation is bound to follow – unless other candidates are downgraded, which is unthinkable. Are autumn exam candidates being set up to fail? Or will the August results be scaled down to allow some headroom in the national distribution?
Furthermore, students sitting autumn exams face a compulsory gap year, because the exams will be too late for a 2020-2021 start. This in itself may be discriminatory, especially for disadvantaged students. The impact of autumn-awarded grades on admission prospects for 2021 is uncertain. Some universities are refusing deferred entry for 2021, others will honour offers but with added conditions. The competition for 2021 entry is likely to be much more intense as 2020 students reapply, a larger 2021 cohort apply for the first time, and international students from 2020 and 2021 return in much larger numbers.
A different, and more immediate, recourse is to a fit-for-purpose appeals process, which the currently-defined process surely is not:
Centres will be able to appeal against the operation of the standardisation model where the wrong data was used to calculate results for learners, or where there was an administrative error in the issuing of results by an exam board.
This is narrow, technical, and – importantly – an appeal against the process, not against the outcome. If a student believes he or she has been awarded the wrong grade, the process by which that grade was determined is neither here nor there: it’s the outcome that matters. And to deny an appeal against an unfair outcome using the defence that a perverse process was conducted fully in accordance with its own flawed rules flies in the face of natural justice.
The Select Committee has made laudable recommendations to make it easier for the less “well-heeled and sharp-elbowed” to appeal, but the grounds for appeal remain technical and in practice impossibly narrow. Far better would be a broader basis for appeals on the grounds that the awarded grade is believed to be unfair, regardless of the wealth, ethnicity or any other characteristic of the candidate. There needs to be convincing evidence, but the principle of such an appeal is fundamental. This would not be an appeal against academic judgment; on the contrary, it would be an appeal to restore the academic judgment of teachers overturned by a statistical algorithm.
There are two nearby precedents. In Scotland, the SQA recently confirmed that the appeals process will be free, and allow for ‘further, evidence-based consideration of grades if schools and colleges do not think awarded grades fairly reflect learner performance’. And the Republic of Ireland has a three-stage process, of which the third is ‘If the student remains unhappy with the outcome after stages 1 and 2 he/she can seek a review by Independent Appeal Scrutineers’.
Looking ahead, there is a danger is that, in trying to meet the Committee’s recommendations, and to preserve the precious national grade distribution, Ofqual will merely tinker with their model to prevent the worst excesses of unfairness to high-profile groups such as socioeconomically disadvantaged and BAME students, with the price being paid by others – who will indeed be discriminated against but are considered more ‘expendable’. We need instead some modest relaxation of the policy of “no grade inflation” to compensate for all the individual unfairnesses.
The consequences would be minimal. For university entry, most universities will be sympathetic because they will be short of students – the trough of the 18-year-old demographic and major loss of international students. For broader employment purposes in the longer term, young people this year and for the next few years are very likely to be disadvantaged by the recession; the exam system need not make it even worse.
The Select Committee’s report is direct, relevant and hard-hitting. But it does not bring this important episode to a close.
When we can see inside the statistical standardisation black box we will know more about its inherent unfairnesses, especially those attributable to an overarching requirement to enforce “no grade inflation” even at the expense of fairness to individual candidates. Those unfairnesses will require attention.
And we need an appeals process which is easily accessible, broader, and – most importantly – fairer.