This blog was contributed by Dennis Sherwood who has previously written about A levels.
At the Education Select Committee hearing on 2 September, Ofqual’s Chief Regulator, Dame Glenys Stacey, acknowledged that exam grades are ‘reliable to one grade either way’.
Two weeks later, at the hearing on 16 September, Gavin Williamson, the Secretary of State for Education, stated that ‘What is clear is that the exam system is the fairest and best system available to us’, a sentiment echoed by Nick Gibb, the Minister for School Standards, when he informed the Committee on 20 October that he believed exams to be ‘the fairest way we have of assessing children’s work, ability and aptitude’.
I am puzzled. To me, these statements are mutually contradictory. If exams are ‘reliable to one grade either way’, does that mean that A level results designated ABB are in fact any set of grades from A*AA to BCC? But if entrance to a university programme requires grades AAB, then any student with a certificate of ABB could well be rejected. How can this be ‘fair’?
Some further light on this has been thrown by the letter from the Committee’s Chair, Robert Halfon, to Gavin Williamson, dated 10 November, tabling some of the conclusions from the recent hearings.
The letter is hard-hitting, for example:
In early September Ministers made assurances to the House and us that all the relevant information and papers would be provided to this Committee, so that we could look into this matter. As yet, the delivery of this information has not materialised, and so we would ask that we receive the papers requested by our clerks by Monday 23 November at the latest.
It was revealing to us that they recognised this problem but simply accepted that they could not find a solution to it and chose to carry on. Ofqual conclude that what they had been asked to do was in fact an impossible task. Yet when the Chief Regulator came before us in June, she did not use the opportunity to raise any alarm bells with us, choosing not to do so publicly at all.
We regret that Ofqual decided not to raise wider concerns about the fairness of the model they were being asked to implement. They had every opportunity to do so when they came before us in June. Instead, they simply followed the ministerial direction and hoped for the best.
This whole episode calls into question Ofqual’s independence, its way of operating and its Board’s duty and ability for maintaining standards.
Each of those merits a blog in its own right, but here I wish to focus on ‘fairness’ and ‘reliability’, in relation to which this extract from Mr Halfon’s letter is most informative:
Ofqual was established under the Apprenticeship, Skills, Children and Learning Act 2009 to preserve the integrity of exam qualifications but it has no specific statutory remit to ensure fairness.
That may come as a surprise, for you might imagine that ensuring fairness to each individual student would be at the very heart of the examination system.
It isn’t. Mr Halfon is correct, for Ofqual is indeed constituted under Section 128 of the Apprenticeship, Skills, Children and Learning Act 2009, as amended by Section 22 of the Education Act 2011, which reads:
The qualifications standards objective
In section 128 of ASCLA 2009 (Ofqual’s objectives) for subsection (2) (the qualifications standards objective) substitute—
(2) The qualifications standards objective is to secure that—
(a) regulated qualifications give a reliable indication of knowledge, skills and understanding, and
(b) regulated qualifications indicate —
(i) a consistent level of attainment (including over time) between comparable regulated qualifications, and
(ii) a consistent level of attainment (but not over time) between regulated qualifications and comparable qualifications (including those awarded outside the United Kingdom) which are not qualifications to which this Part applies.
The obligation to ensure ‘a consistent level of attainment over time’ underpins the ‘no grade inflation’ policy, and the ‘consistent level of attainment [over space]’ is intended to ensure equality of academic standards.
The concept of ‘fairness’, however, is not mentioned. The possibility that some students might be treated unfairly by the exam system, or by this summer’s algorithm, therefore appears, from a legal point-of-view, to be neither here nor there. Presumably, any appeal based on grounds of unfairness would be dismissed out-of-hand. It seems as if school exams don’t have to be fair.
Which, indeed, they are not – for a host of reasons, not least the unreliability of the outcomes, the grades indelibly printed on candidates’ certificates.
But is there a catch? Although Section 22 does not talk of ‘fairness’, it does place a duty on Ofqual to ‘secure… a reliable indication…’.
In this context, what is the significance of Dame Glenys Stacey’s statement that exam grades are ‘reliable to one grade either way’? Is this a ‘smoking gun’, conclusive self-confessed evidence of a breach of the Act? Or would a clever lawyer argue that Section 22 does not specify what degree of reliability should be secured, and that ‘reliable to one grade either way’ is sufficient, and fully compliant with Ofqual’s legal obligation?
I’m not a lawyer, so I don’t know.
But as a member of the public, I have a view.
To me, exam grades should be fair, and a key aspect of ‘fairness’ is reliability. And grades that are ‘reliable to one grade either way’ can never be fair.
This is especially relevant in the context of the recent recommendations from both UCAS and UUK that universities should make admissions offers only after the A level results have been announced, so as ‘to make the system fairer by eliminating the use of predicted grades, which are often unreliable’.
In making these recommendations, UCAS and UUK appear to be placing even more importance on exam results, endorsing and trusting their reliability. Such trust, however, is misplaced. On average, about 1 grade in every 4 is wrong, with significant variations by subject (about 1 grade in 25 is wrong in Maths, nearly 1 in every 2 in History), and also by mark within subject (in essence, a script marked at or close to any grade boundary, in any subject, has a probability of about 50% of being awarded the right grade, so tossing a coin would be as fair).
Making offers after the results have been announced, when those results are so unreliable, doesn’t seem to me to achieve much. So perhaps UCAS and UUK should be demanding that Ofqual deliver grades that are not ‘reliable to one grade either way’ but ‘reliable, full stop’.
One defence, of course, is for Ofqual to claim that there is no credible alternative. This, however, is not the case, for it is very easy to award assessments that approach 100% reliability. But to do so requires a change in the policy by which the assessment shown on a certificate is determined from the corresponding exam’s marks, and also a change in the policy for appeals.
The pity is that Ofqual appear to be very reluctant to contemplate doing this.
Perhaps the current turmoil concerning the plans (or otherwise) for exams in England in summer 2021, along with Robert Halfon’s letter and the resulting fall-out, as well as the reactions to the recent proposals from UCAS and UUK, will increase the pressure on Ofqual for beneficial reform.
I do hope so.
Knowing that ‘one in four grades is wrong’ and that most are reliable to one grade either way should be no kind of reassurance to anyone.
When we hear that, we should remember who that one in four is. They might be a student whose life takes a path different to the one they had hoped because they got a D rather than a C. We should think of their disappointment, the opportunities that might be closed to them as a result and then the further opportunities that will close too because they missed the first. Meanwhile, all those privileges are being enjoyed by someone whose grade was undeservedly bumped up.
We should remember that it was not because their work missed the grade: it’s because their grade did not reflect their work. An examiner’s decision (or an algorithm) made a mistake.
It should not matter whether this injustice is happening to one in four or one in twenty-five, the fact that it is unjust and that Ofqual, DfE and universities accept it is cruel.
That said, we know that university departments need to look not only at the individual when they approach admissions, but also at the cohort in aggregate. So long as they can try to minimise the injustice that they perpetrate themselves or that they perpetuate in others, universities can certainly claim to be acting with integrity and in the best interests of students.
After all, what else do they have to go on? What else might eliminate injustice? Interviews? Their own exams? We know that these mechanisms favour students from particular backgrounds.
I was struck by something Anne-Marie Canning (now CEO of the Brilliant Club) said recently. She was talking about the approach they had engendered when she was in her previous role as Director of Widening Participation at King’s College London. Rather than considering whether candidates were good enough, they looked at WP students’ applications with the mindset of ‘What can we do to make this student an offer?’ That struck me as an approach that allows the admissions process to recognise that perhaps a grade might be wrong, perhaps a prediction might be pessimistic, or even perhaps that although the grades might reflect a particular student’s achievements, it is their achievements that do not reflect their potential.
So the system can be said to be working ‘well enough’ only if, first, we ignore the evidence that it isn’t and, second, we take steps to counteract the unreliability of the grades by introducing flexibility and contextualisation (and recognise that, in doing so, we might introduce yet further biases and injustices).
There is much talk of A levels as ‘a gold standard’ and their ineluctable significance would be gilded still further by the Government’s proposals last week for PQA – and, to a lesser extent, by UUK’s and UCAS’s proposals for PQO. It feels terribly precarious to balance so much on such shaky foundations.
I would be interested to know more about how exams could be more reliable but I do fear that, what with the need to tear up A level grading, the multiple calls for admissions reform, the introduction of T levels and dumping of BTECs, the shocks to the system that Covid has wrought on GCSEs and other secondary exams across all UK nations, we’re playing a hazardous game of Jenga. The fundamental blocks are rocky and every change we make to adapt for the future or stabilise what exists just makes the whole edifice more shaky.
It’s a shame that New Labour lost its nerve when it had the chance to overhaul post-14 education, but even their Tomlinson proposals would no longer be fit for today.
Dennis’s excellent blog and tireless interrogation of the issue of A level grading highlights that A levels are a far cry from the qualifications we now need – too unfair, too narrow, too old-fashioned – and that’s because they reflect a secondary education system that, essentially, has barely changed in many decades even while HE progression has become a mainstream, the compulsory education age has increased, the need for technical pathways has grown and the entire labour market has changed.