- This guest blog has been kindly written for HEPI by Dennis Sherwood, the author of Missing the Mark – Why so many school exam grades are wrong, and how to get results we can trust, (Canbury Press, 2022).
GCSE, AS and A level students will soon be sitting their exams. Their goal will be the highest possible grades, for they know that good grades open doors; poor ones slam them shut, possibly for ever. ‘High stakes’ indeed.
If you’re involved in admissions, grades are important to you too. You may have used data on historic grade distributions to determine your criteria for entry, and you might use candidates’ actual grades to decide who gets in, and who doesn’t. Depending on the programme, and the matching of supply with demand, there may be some flexibility, but for highly competitive courses, such as medicine, there will be a sufficient number of applicants with certificates meeting the offer threshold of, say, AAA for A level Biology, Chemistry and Physics to fill all the places, so that student with AAB will be rejected.
Here’s a statement that might be significant:
It’s really important that people don’t put too much weight on any individual grade.
That’s very general. ‘People’ embraces students, parents, admissions officers. And the stress on the importance of ‘not putting too much weight on any individual grade’ is, as I will show shortly, code for ‘If a student has a certificate showing grade B for A level Physics, that B can’t be trusted – perhaps the student truly merited a C. Or even an A.’
That’s troubling. For it implies that a certificate showing AAB might be in error, and should have shown AAA. In which case, that student was mistakenly rejected. And it also implies that a student whose certificate shows AAA, and therefore accepted, might have truly merited AAB, and so should have been rejected.
Oh dear.
I didn’t make that statement up.
They are the words of Dr Michelle Meadows, Associate Professor in Educational Assessment at the Department of Education at Oxford, and formerly Ofqual’s Executive Director for Strategy, Risk and Research. And although the context concerned GCSE grades (Dr Meadows was giving evidence at the hearing of the House of Lords Education for 11-16 Year Olds Committee held on 30 March 2023 – pick up at 11:57:55 ), the statement applies just as forcefully to AS and A level grades too.
My hunch, however, is that many people – students, teachers, and possibly admissions officers as well – don’t act as Dr Meadows advises, but do indeed place considerable weight on individual grades. The difference between AAA and AAB is the opportunity to become a doctor, or not; grade 3 rather than grade 4 in GCSE English implies a total disruption of the normal progression of education, and a requirement to re-sit the exam, with who-knows-what consequences on the student’s mental health.
The statement that ‘It’s really important that people don’t put too much weight on any individual grade’ is the most recent in a series of utterances from authoritative sources hinting that school exam grades are not as reliable as you might expect them to be. Here are some others:
The probability of receiving the ‘definitive’ qualification grade varies by qualification and subject, from 0.96 (a mathematics qualification) to 0.52 (an English language and literature qualification). The probability of receiving the definitive grade or adjacent grade is above 0.95 for all qualifications, with many at or very close to 1.0 (ie suggesting that 100% of candidates receive the definitive or adjacent grade in these qualifications). (Ofqual, November 2018, Page 4, here)
…more than one grade could well be a legitimate reflection of a student’s performance and they would both be a sound estimate of that student’s ability at that point in time based on the available evidence from the assessment they have undertaken. (Ofqual, 11 August 2019, here)
There is a benchmark that is used in assessment evidence that any assessment should be accurate for 90% of students plus or minus one grade. That is a standard benchmark. On average, the subjects were doing much better than that. For A-level we were looking at 98%; for GCSE we were looking at 96%, so we did take some solace from that. (Dr Michelle Meadows, 2 September 2020, Q997 here)
It is interesting how much faith we put in examination and the grade that comes out of that. We know from research, as I think Michelle mentioned, that we have faith in them, but they are reliable to one grade either way. We have great expectations of assessment in this country. (Dame Glenys Stacey, 2 September 2020, Q1059 here)
Ah. Grades ‘are reliable to one grade either way’. So that’s why ‘it’s really important that people don’t put too much weight on any individual grade’.
In real life, however, people put huge weight on those individual grades. They matter.
So here’s a question for all those involved in admissions.
Suppose, this August, a student phones you and says, ‘I have an offer of ABB. My actual grades are ABC. According to Ofqual, grades are only reliable to one grade either way. May I have my place, please?’
What might be your response?
If you’re about to suggest ‘You should appeal right away’, let me point out that Ofqual’s appeals process deliberately denies the discovery of grade errors attributable to differences in the academic opinion of examiners – which to me is somewhat puzzling given that this is the fundamental reason why grades are only ‘reliable to one grade either way’.
An appeal therefore cannot determine whether or not the ABC should have been ABB, and so the question ‘May I have my place, please?’ remains, as does the need for a response.
Some further information might be helpful. Ofqual’s measurements of the reliabilities of grades for various subjects, as shown in Figure 12 on page 21 of their November 2018 report Marking Consistency Metrics – An update, and as discussed here, imply that for every 100 students whose certificates show AAA for Biology, Chemistry and Physics, about 17 merited at least one B. That’s about 1 student in 6. And a similar number with at least one B (and no Cs) truly merited AAA. For English Language, English Literature and History, the numbers are rather different: for every 100 students ‘awarded’ AAA, about 50 merited at least one B. That’s around one-half. And about 50 in every 100 with at least one B (and no Cs) merited AAA.
That phone might be ringing quite frequently.
Is the situation made worse by the fact that students are being judged on only 3 subjects? That system, imposed on schools for decades and decades solely for the purpose of university selection, has already done wondrous damage to the post-16 curriculum. If it’s no good for choosing candidates, either, what’s the point of it all?
Should the question be: why give entry to an AAA ahead of BBB?
Can you show BBB is not suitable to do the course?
If not would fairer equal random selection of all at whatever threshold is deemed tied to being ready for the course?
If grades are only reliable to one grade either way then they are not fit for how universities are using them in their admission processes. Is it Ofqual that is in the wrong for presiding over grades that are not fit for that purpose, or is it the universities that are in the wrong for using grades in a way that Ofqual does not intent? Could Ofqual be guilty of not being open enough about the fact that its grades are only reliable to one grade either way?