This blog was kindly contributed by Rob Cuthbert, Emeritus Professor of Higher Education Management, University of the West of England. You can find Rob on Twitter @RobCuthbert.
Tomorrow at 10am, HEPI will be hosting an ‘In Conversation’ session with Clare Marchant, Chief Executive of UCAS. Details here.
As A-Level results day (Thursday, 13 August) looms, there is still widespread misunderstanding among students and parents about how grades have been determined.
If students are dissatisfied with their grades there is probably nothing they can do about it, except take exams in the autumn. Contrary to what many people think, most grades are not based on teacher assessments, and the prospects for appeals against an unfair grade are limited and uncertain. Autumn exams probably mean an enforced gap year, and 2020 university offers may not hold for 2021.
1. Most grades are not based on teacher assessments
The Secretary of State announced on 20 March that there would be no exams in 2020 and ‘The exam boards will be asking teachers … to submit their judgement about the grade that they believe the student would have received if exams had gone ahead.’ He added there would be controls ‘to ensure that the distribution of grades follows a similar pattern to that in other years, so that this year’s students do not face a systematic disadvantage as a consequence of these extraordinary circumstances.’ In fact, it is this year’s students who face disadvantage from these ‘controls’.
Examinations are conducted under the auspices of a number of examination boards, all regulated by the national agency Ofqual. Ofqual consulted schools, teachers and parents before outlining the approach it would require exam boards to follow. its most radical innovation was the requirement for schools not only to propose grades for each student but also within each grade band for each subject to rank all students in order – with no ‘second equals’ or similar rankings.
The approach soon attracted fundamental questions about how it would work. The House of Commons Education Select Committee inquiry into the broad effects of COVID-19 on the education system received so many submissions about exams in 2020 that it published an interim report while it was still taking evidence.
The interim report asked Ofqual to publish details of its approach ‘immediately’. Ofqual refused: Chief Regulator Sally Collier was forced to admit that publishing its model would also enable schools to calculate awarded grades immediately. Under pressure, Ofqual ran a Summer Symposium on 21 July where it admitted for the first time that, having tested 12 different approaches, it would use what it called the ‘Direct Centre-level approach’.
For ‘small’ cohorts the Ofqual method will substitute Centre (school) Assessed Grades (CAGs) for the results which the model would otherwise require. Ofqual did not define ‘small’ in this context until 7 August, when the Times Educational Supplement reported that:
Where a subject has more than 15 entries in a school, teachers’ predicted grades will not be used as part of the final grade calculation … Where there are subjects with no more than five entries in a school … pupils will be awarded their teacher-assessed grades … For entries between five to 15 students, teacher-assessed grades will play a role in the calculation, alongside historic school data and pupils’ prior attainment
What this means is that, as has been suspected for some months, teacher assessments of grades are largely irrelevant. The only thing that counts is the ranking.
School teachers, like academics in higher education, do not make judgments about whether a student is 12th or 13th best in a group of 20 all assessed as a B. Academic judgment is about whether student X deserves a certain grade or not – it focuses on the boundaries between grades, and draws on teachers’ understanding of what each grade achievement means. It does not predetermine the numbers in each grade; it determines which grade each student deserves.
That is why the Ofqual approach was so controversial; it asked teachers to do something they had never been asked to do before, and for the first time it set students in competition with their fellow-students for the limited number of grades allowed to each school. Even worse, it judges students not on their merits, but on the merits of previous cohorts from the same school.
Government and Ofqual have represented the approach as being based on ‘teachers’ assessments of grades’, when it is not. As independent analyst Dennis Sherwood wrote on 18 June: ‘It would have been both more honest, and far simpler, for each board to have written to each school saying “our statistical algorithm has determined that, for your cohort of 53 students for 2020 A level Geography, you are allocated 4 A*s, 10 As, 15 Bs … Please enter in each grade box the names of the students to be awarded each grade, ensuring that no grade exceeds its allocation”‘.
2. Schools might delay telling you their predicted grades and rankings
Ofqual have said that schools may tell students and parents their proposed grades. As for rankings, they might or might not be revealed, depending on whether revealing the ranking for the student concerned might also reveal personal ranking data about other individuals. It is for each school to decide its policy, but individual students or parents may submit a Subject Access Request or a Freedom of Information Request (FoIs apply to public bodies like state schools) to get proposed gradings, and perhaps ranking, information. Such requests must receive responses within one month from the date the request is received – which presumably means the response might not be received until 13 September. In many cases this would be too late for university entry, especially for selective universities which may complete admission decisions within days of the results announcement.
3. The prospects for appeals against unfair grades are limited and uncertain
Government and Ofqual statements make it seem that the appeal process will protect students against grades they do not deserve. But the process as first defined was far too limited:
Centres will be able to appeal against the operation of the standardisation model where the wrong data was used to calculate results for learners, or where there was an administrative error in the issuing of results by an exam board.
This allowed for only narrow, technical appeals against the process, not the outcome, as Dennis Sherwood and I wrote in a blog for HEPI on 14 July:
To deny an appeal against an unfair outcome using the defence that a perverse process was conducted fully in accordance with its own flawed rules flies in the face of natural justice.
Far better would be a broader basis for appeals on the grounds that the awarded grade is believed to be unfair, regardless of the wealth, ethnicity or any other characteristic of the candidate. There needs to be convincing evidence, but the principle of such an appeal is fundamental. This would not be an appeal against academic judgment; on the contrary, it would be an appeal to restore the academic judgment of teachers overturned by a statistical algorithm. [emphasis added]
On 6 August 2020, Ofqual issued its ‘final guidance’ on appeals, adding some new ‘examples’ of when an appeal might be made, in particular:
where – because of the ability profile of the students – a centre was expecting results this year to show a very different pattern of grades to results in previous years. That could include where the grades of unusually high or low ability students been affected by the model because they fall outside the pattern of results in that centre in recent years.
This may, in practice, prove too high a hurdle for most schools to overcome, and in any case it still does not provide for any individual appeals, especially where an individual has been unfairly treated within a cohort of similar ability to previous years. This minor change was presented in the media as a major concession, even though Ofqual itself has said it expects such appeals to be rare. At the same time Ofqual implied in its 6 August statement that there had been no change beyond minor modifications to its original decision about the appeal process. With no statement on appeal fees, it must be assumed that fees will remain in place, providing a further hurdle for appellant schools.
Other approaches are possible. For students in the UK taking the International Baccalaureate (IB) in 2020, the IB board received a deluge of criticism when its results were announced on 6 July, and instituted an apparently comprehensive appeals process. The Republic of Ireland has a three-stage process, of which the third is ‘If the student remains unhappy with the outcome after stages 1 and 2 he/she can seek a review by Independent Appeal Scrutineers’.
In Scotland, the SQA confirmed the appeals process will be free, and will allow for ‘further, evidence-based consideration of grades if schools and colleges do not think awarded grades fairly reflect learner performance’. However, appeals must be submitted by 14 August, putting school staff under even more pressure at a time when they are making arrangements for a socially-distanced return to school for next year’s students.
When Scottish results were announced on 4 August there was immediate disquiet about the outcomes, particularly for socioeconomically disadvantaged students. Among the 138,000 students there were 125,000 reductions in grading – about a quarter of all grades.
First Minister Nicola Sturgeon defended the SQA approach by claiming the appeals process would address any individual unfairness and that to accept schools’ proposed grades would have led to unacceptable grade inflation. Perhaps the SQA’s statistical model has indeed simply replicated existing disadvantage, as David Kernohan suggested in his analysis on 5 August 2020 for Wonkhe. A growing storm of criticism of the SQA approach included a devastating critique from Guy Nason, a statistics professor at Imperial College, which ended:
The problem at the heart of the statistical standardisation is that it can be simultaneously unfair to individuals, but also maintain the integrity of the system. However, if system integrity damages the life chances of individuals, then it is not much of a system.
The undue focus on collective fairness has led to neglect of fairness for individuals. While the examination system throws up many such issues every year, in normal years there is at least both an actual individual performance to assess, and an appeal process. This year there is neither. A previous HEPI blog said:
there is a danger that, in trying … to preserve the precious national grade distribution, Ofqual will merely tinker with their model to prevent the worst excesses of unfairness to high-profile groups such as socioeconomically disadvantaged and BAME students, with the price being paid by others – who will indeed be discriminated against but are considered more ‘expendable’. We need instead some modest relaxation of the policy of ‘no grade inflation’ to compensate for all the individual unfairnesses.
It is understandable that in the rush to introduce a completely new system it seemed reasonable at first to invent a system in which dissatisfaction could be tackled by an opportunity to take an autumn examination. Time and others’ experience has shown this to be mistaken. If initial results match the allowed national distribution and autumn exam candidates succeed in achieving higher grades, then grade inflation is bound to follow – unless (unthinkably) other candidates are downgraded. Are autumn exam candidates being set up to fail? Will August results be scaled down to allow some headroom in the national distribution?
Furthermore, students sitting autumn exams will almost inevitably face a gap year, because the exams will be too late for students or universities to manage a 2020/21 start. This in itself may be discriminatory, especially for disadvantaged students. The impact of autumn-awarded grades on admission prospects for 2021 is uncertain. Some universities are refusing deferred entry for 2021, others will honour offers but with added conditions. The competition for 2021 entry is likely to be much more intense as 2020 students reapply, a larger 2021 cohort apply for the first time, and international students from 2020 and 2021 return in much larger numbers.
4. An autumn exam probably means a gap year; 2020 offers might not apply in 2021
Sally Collier of Ofqual has urged higher education providers to focus less on this year’s calculated results and instead place more weight on other evidence, such as speaking to the student’s school to assess their potential. It would be better for Ofqual to take some responsibility for the individual unfairnesses inherent in their model and give more weight to teachers’ proposed grades in the first place. The consequences for university entry would be minimal: most universities will be sympathetic, not least because of the trough of the 18-year old demographic and major loss of international students. For broader employment purposes, young people this year and for the next few years are very likely to be disadvantaged by the recession; the exam system need not make it even worse.
5. Won’t most students get the grades they deserve anyway?
At this stage, no one knows for sure, but the chances are slim. The key slide in Ofqual’s Summer Symposium has been heavily criticised for its inconsistencies and one analysis based on Ofqual data suggests that 39% of centre-assessed grades will be downgraded while 61% will remain the same. Higher education data guru Mark Corver has been calling, in the interests of fairness, for UCAS to publish information showing the spread of actual grades against UCAS predicted grades (not the same thing as this year’s CAGs), which would give a different way of assessing which A-Level grades students deserve. UCAS have refused on the grounds that this might enable identification of individuals in some cases.
At a national level, there will be sophisticated analysis of how far this year’s grades deviate from the norm established in recent years, but individual unfairnesses are likely to be legion. The Ofqual approach applies at the level of every grade in every subject in every school. Ofqual data show that the average year-on-year standard deviation between the percentage of grade A and above for large cohorts (defined for this purpose as 20 or more) in one subject in one school is of the order of 12-14%. How much statistical confidence can we have about results from applying the algorithm to every group of students in one grade in one subject in one school?
Attempting to spin the narrative, Ofqual blogged on 6 August about ‘;Fairness in Awarding‘.
Exam board technical teams have worked with us to develop and test various statistical models to standardise teachers’ judgements between centres, to make sure that the one we used is the fairest possible approach, and we’ve also had input from external statisticians and assessment experts. … Of course, a system of calculated grades and a statistical model can never know how an individual student might have performed on the day. Some students might have done better, or worse, if they’d had an opportunity to take their exam; we will never know and they will never know.
This appeal to ‘expertise’ and a ‘statistical model’ (which we have to take on trust, since it remains secret until results day) completely misses the point. The obsession with ‘fairness’ at national level means thousands of individual students will get grades they do not deserve, whether higher or lower.
For the first time ever, students are competing within the school with their schoolmates for the limited numbers of grades available at each level. These are not the grades this year’s students deserve, they are the grades which the Ofqual model says the school deserves on the basis of past students’ achievements.
The only way to restore individual fairness is to restore individual appeals which look at the student’s actual achievements, not the past record of the school. Every previous student has had, every future student will have, access to an individual appeal process. Not in 2020. So much for the Secretary of State’s pledge that this year’s students should not face ‘a systematic disadvantage as a consequence of these extraordinary circumstances’.
Ofqual seem to have arrived in the worst of all worlds, with secrecy, schools working out how best to use a system they don’t understand, limited appeals or checks on school submissions and a scramble to avoid penalising disadvantaged students. But in a zero-sum game this means penalising what the Select Committee called the ‘well-heeled’ middle classes with, quite possibly, a well-heeled slew of litigation to follow. If as Ofqual admit, ‘most students will have one grade changed’ then there is virtually no room for manoeuvre for anyone with, say, an Oxbridge offer of A*A*A, or one HEPI blog commentator whose daughter has an Imperial College offer of A*A*A*A.
The Scottish government has got it badly wrong over its exam results. England, Wales and Northern Ireland’s governments should learn from its mistake ahead of their own exam results on 13 August. … Some people have received worse results than they would have got had the exams taken place, while others have benefited. The aggregate results are more ‘fair’ yes, but our lives aren’t lived in the aggregate. The people who have had their results downgraded are experiencing a great unfairness.
In some parts of the media the narrative has changed, with commentators now blaming schools for overbidding. Ofqual have so far facilitated this without quite blaming schools, saying ‘it’s not surprising’ that schools want to do the best they can for their students. But if it’s not surprising, Ofqual should have anticipated it and changed their approach, at least by allowing appeals. As it is, students and parents can only wait with trepidation to discover whether what the national authorities regard as fair will be fair for them.