- This HEPI blog was kindly authored by Rob Cuthbert of the Practical Academics consultancy.
Let us assume that higher education institutions want to have the best students they can get, admitted in the fairest way possible: not a very controversial assumption, even if sometimes people look to satisfice, by settling for something good enough, but perhaps not best. The assessment of ability and potential to succeed in most English universities is based to an important extent on the grades achieved at A-level, complemented by other factors whose importance varies with context. Universities use grades, often in a fine-grained way, to set conditions for admission and then to discriminate between applicants of apparently similar standing, perhaps driving school attainment but certainly helping the universities to manage their admissions to minimise or prevent over-recruitment.
Research by Dennis Sherwood (independent), discussed in a recent HEPI blog has shown that grades for GCSE and A-level examinations are unreliable: one grade in four is wrong. However good the marking of exams may be, there is inherent ‘fuzziness’ in all subjects, which has led Ofqual to admit that ‘more than one grade could well be a legitimate reflection of a student’s performance and they would both be a sound estimate of that student’s ability at that point in time based on the available evidence from the assessment they have undertaken.’ (Ofqual, 11 August 2019). Michelle Meadows (Oxford), formerly Ofqual’s Executive Director for Strategy, Risk and Research, in evidence to a House of Lords Committee on 30 March 2023, said: ‘It’s really important that people don’t put too much weight on any individual grade.” Ofqual head Glenys Stacey said in 2020: “It is interesting how much faith we put in examination and the grade that comes out of that. We know from research, as I think Michelle mentioned, that we have faith in them, but they are reliable to one grade either way.’ (Dame Glenys Stacey, 2 September 2020, Q1059)
Sherwood’s analysis seems irrefutable, and it should have profound consequences for admissions. Prized university places may hinge on one difference in one grade, but all grades are only ‘reliable to one grade either way’. If one grade in four is wrong, the chance of one student getting three correct grades is on average 27/64. However the average conceals wide variation, with STEM subjects less ‘fuzzy’, whereas in English, for example, only about half of all grades are correct. Ofqual has repeatedly rejected Sherwood’s analysis and indulged in ad hominem attacks, without refuting his arguments. Worse, Ofqual has also reduced the possibility of successful appeals against grades to a negligible level. Changes to grading practices to deal with the unreliability are feasible and might even be straightforward, but Ofqual has shown itself unwilling to consider such changes. It has welcomed the reversion to pre-Covid grade distributions, but that makes no difference to the underlying unreliability of grades. It follows that higher education institutions must do what they can to ameliorate the unfairness which inevitably follows. How should admissions practices and criteria change?
There is a range of possibilities. First and simplest, albeit broad-brush, would be to accept the reality of unreliable grades by announcing that all applicants will be considered in terms of their grades plus one grade. A BBC applicant would be considered as AAB, and so on. (This would be a more drastic response than Sherwood says is necessary, but Ofqual refuse to make the necessary lesser changes.) That change in admissions practices would give all the benefit of the doubt to applicants (save for a tiny minority who might have merited two grades better). It would be fairer to allow the possibility of grades being lower as well as higher: this is also achievable. For example, adopt a formula which allows any one grade to be increased by one, and any other grade to be reduced by one, with the third unchanged. Another approach would be to accept grade changes for only the ‘fuzziest’ subjects (such as English and History). For example, simply consider grades plus one for a defined list of subjects, or all subjects except STEM. There are of course many other possible formulae, but the simplest, and least likely to lead to objection or complaint, is the first. It would overstate likely achievement, and it would add to congestion at the upper end of the range of grades, but it would ultimately be fairer than the lottery of the existing grading system.
The unavoidable consequence would be compliance with Michelle Meadows’s recommendation not to put too much weight on any one grade. This would cost universities more. Admissions officers would be bound to give more weight to other considerations, such as universities’ own admissions tests, the UCAS personal statement and more. Interviews might become more widespread. Universities would have to invest more in the selection process, while perhaps reducing their marketing spend in other directions. There would also be logistical complications. The timescale for considering applications might have to be extended, perhaps to match the longer timescale already in operation for applications for Medicine courses, and for Oxbridge. These are significant issues with resource implications, but the alternative is to continue in the unfair but convenient fiction that grades as awarded are an accurate reflection of performance, ability and potential.
Would this change be fairer? For some courses at some institutions, applicants are even now admitted after results day or in Clearing with lower grades than at first required. To a considerable extent, institutions might cope simply by adjusting grade requirements, but would now be less concerned that the lottery of the existing grading system had eliminated promising applicants. The highest grades would be more widespread; the long-term effect is likely to be fairer to most disadvantaged applicants – they might be more inclined to aim high, and they might be more likely to pass a grade-linked threshold for consideration. The change might reduce the need for contextual offers, since many such offers would not allow for more than one grade below the standard offer. All institutions would be driven to give more weight to other information about applicants, and perhaps to rely more heavily, for example, on partnerships with schools and colleges, which will be very well-informed about the potential of their higher education applicants and their ability to succeed. Professor Stephen Jones (Manchester) and his colleagues have recently argued for changes to the UCAS personal statement because their research suggests it is unfairly discriminatory, especially against applicants from disadvantaged backgrounds and / or less supportive schools. The jury may still be out on some points, but UCAS have shown their willingness to consider changes, and consulted on which changes to make.
What of applicants who do not rely on A-level grades? Evidence on the grade reliability of Scottish highers, the International Baccalaureate and the range of BTEC and newer technical qualifications is much less clear-cut, so this would be problematic, but it might be safer (or fairer) at first to allow a similar formulaic dispensation for those qualifications too.
There is plenty of evidence that some categories of student may suffer disadvantage in admissions because of the grades they achieve, if for example they come from some ethnic minorities, if they have lower socioeconomic status, if they have disabilities, and so on. And there is plenty of evidence that institutions work hard to overcome such disadvantages. By changing the way institutions treat grades for admission purposes one source of unfairness should be alleviated. Others will remain, but come under much greater scrutiny because the myth of examination grade fairness will have been scotched. If Ofqual refuse to change their grading practices, universities must change their admission practices. Things can only get better.
If you enjoyed this article, you can also read Rob Cuthbert’s previous HEPI blogs: How to make HE admissions fairer, quicker and better and A-levels 2020: what students and parents need to know.
HEPI recently produced a presentation aimed at Year 12 and Year 13 students considering their options. See here.
” First and simplest, albeit broad-brush, would be to accept the reality of unreliable grades by announcing that all applicants will be considered in terms of their grades plus one grade. A BBC applicant would be considered as AAB, and so on. (This would be a more drastic response than Sherwood says is necessary, but Ofqual refuse to make the necessary lesser changes.) That change in admissions practices would give all the benefit of the doubt to applicants (save for a tiny minority who might have merited two grades better).”
This doesn’t really make sense. Many courses have limited capacity. Some (e.g. medicine) are tightly capped; some lab sciences can’t go beyond a certain number of students because it would become impossible to timetable lab classes or would require the course to change to have less lab activity, making it less valuable. This is before you even consider that most universities already have insufficient accommodation for the number of students they accept. Accepting BBC students onto a course with an AAB offer would mean making fewer offers, so there would still be equally qualified students being excluded.
It is deeply ironic that 3 A level grades should turn out not to be fit for purpose in creating a fair process of selection. After all, schools have spent decades teaching the narrowness of 3 subjects precisely because that is the system that was invented and imposed on them by universities. Many voices of reason, including The Royal Society and the Times Education Commission, argue for a broader 16+ curriculum and that broader curriculum, as in the International Baccalaureate, does at least offer more data, and greater flexibility, by which a candidate can be judged.
All in that first assumption really.
Counter assumption – that we will find a suitable place for each person who applies.. then the issue is only what does the person know, what can they do now, what potential to learn?
Should we be using A levels and other academic qualificatrions at all, to decide who will be given entry to a course at a University?
Not everyone with good A level results gets equally good grades for their degree.
The degree grades awarded by Universities are of even less fairness / accuracy / value / use than A levels in helping employers pick the best future employees. This is because each University / faculty / department use different criteria and have different curricula for the same subjects.
As they say in the world of stocks and shares “Past performance is no guarantee of a similar future.”
Currently we live in a fantasy world that believes past academic success at school is the most sensible way to predict future performance at University and academic success at University indicates who will be best for a job.
The skills required to thrive in a school, University, job are very different, particulary Uni to Job.
Life is a lottery. Get used to it.
The issue of the (un)reliability of grades featured in a hearing of the House of Lords Education for 11-16 Year Olds Committee held on 8 June, the same day as this blog – click in at about 12:08:35: https://parliamentlive.tv/event/index/de8de2aa-4a66-4037-85ad-d5c0e6f11291
As you will hear, unreliable grades don’t matter much for aggregate statistics – so a statement like “80% (or whatever) of students with A or A* at a A level get a 2:1 or first” is probably OK. But unreliable grades do much damage to individuals, and although life might be a lottery in general, my view is that HE admissions should not be a lottery, especially a lottery so careless as that of the current grades.
It seems to me that there are three options:
(1) Ignore this, and continue as the past. The only victims are those unfairly rejected (see below), but this has happened for years, so who cares?
(2) For HE admissions to recognise that grades are unreliable by adopting a fair policy, such as described in Rob’s piece.
(3) For HE to put pressure on Ofqual to deliver reliable assessments, which is easy to do; reliable grades that students can trust and that HE can use, alongside other information, to determine admissions.
Here are some numbers to bear in mind.
This August, for every 100 students awarded AAA for Chemistry, Physics and Biology, about 17 should have been awarded at least one grade B. If those 17 candidates are admitted on the basis of AAA, they are not only under-qualified, but have denied places to those 17 candidates who were awarded at least one grade B (but no Cs), but who merited AAA.
For English Language, English Literature and History, the corresponding number is about 50 in every 100.
And bear in mind too that this all happens even if there are no “marking errors”: these grading errors can therefore never be detected or corrected by the current appeals process.
Again perhaps articulate out, rather than assume, why in a mass system of HE, it is right to select a person with AAB ahead of someone with ABB?
If the world works better with each person achieving the education knowledge and skills they can – it makes no difference which goes where, if both are capable of the course?
Hi Conor… no, it is not “right”; but at the moment, it “is”. And that “is” is not only failing to do what it is intended to do, but is also having the so-called “unintended consequence” of doing great damage.
I find it helpful to distinguish between “what does a wise admissions process look like?”, and “how do we make the current process materially better?”
The former is ‘bigger picture’ and more fundamental; the latter is pragmatic, gives immediate benefit, and could be a first step along the more visionary road.
So although it may not be “right” for a candidate with AAB to be selected over one with ABB, it happens, and many students believe that it happens. If indeed it does happen, then those grades AAB and ABB need to be fully robust, reliable and trustworthy. Which they are not.
As Sir Jon Coles said at yesterday’s Lords Committee meeting: “this is very significant for the individual”. Indeed.
Gordon, yes, the capacity of most if not all courses is limited. The point is that the way some universities limit access is bound to be unfair to some candidates, given the way A-level grades are currently determined. Inevitably some well-qualified candidates will not gain access to their preferred course. The point, however, is to make the selection process as fair as it can be: universities can do more than they do now – if Ofqual won’t do what they could. My strong preference would be for Ofqual to change its practices so that A-level grades would be more reliable. This is preferable to universities doing their own thing, which may well import many more unfairnesses. Note that this isn’t about marking, which may well be as good as Ofqual says it is, it is about grading and what Ofqual does to convert marks to grades.
Well, it’s not Ofqual that converts marks into grades: that’s the exam boards, but they have to conform to a standard approach that is inherently flawed. Any system with cliff edges that mean a 1-mark difference can distinguish between an A and a B while scores within a 15-mark range might all result in a B doesn’t have any value. (Two students with respective grades of A and B might be one mark apart or 30 marks apart.) The same goes for degree classifications, which are also unfair and largely meaningless.
We don’t currently have any tools that allow fair and direct comparison between applicants, with the possible exception of the aptitude tests sat by medicine & dentistry applicants (which have their own drawbacks). The small number of universities that make unconditional offers to students who pass their bespoke entry exams might be acting more fairly, as long as they allow the exams to be sat in locations that are easily and cheaply accessible and the exams themselves reflect ability rather than coaching.
The marks become grades but then they are converted into points?
University publish a points range they require prospective students to achieve for different courses to be awarded a place, do they not?
What is the exchange rate for converting grades to points?
Where do interviews fit into the process Universities use when offering a place?
Regardless of the unfairness at the marking stage, converting marks to grades, how many students end up being accepted/refused as a result of other factors such as the way “adjustments “ are made because of social/ background factors that are taken into account for individual students?
Perhaps we are getting over concerned about the importance of the lack of transparency/ accuracy of grades
To respond, if I may, to Gordon and Albert…
Firstly, although it is technically true that the boards set the grade boundaries, the reality is that this is strongly influenced by Ofqual – that has to be the case, for that is the only way that Ofqual can control grade inflation. Ofqual has all the power.
Secondly, there is a one-to-one mapping of A level grades to UCAS points, so A* = 56, A = 48… (https://www.theuniguide.co.uk/advice/ucas-application/new-ucas-tariff-everything-you-need-to-know).
Thirdly, universities can take any number of things into consideration when making an offer, but that offer ends up as grades – ABB – not “ABB or thereabouts, and a good report from the Duke of Edinburgh scheme” or whatever. That ABB might be ‘lenient’ in that other students for the same programme might have been told AAA, but the end game is the same, grades. If that student actually gets BBB or BCC, they may still get in, at the discretion of the admissions officer. But to the student, that offer of ABB is hard-edged, and can be used as a reason for saying “sorry” to the student with BBB. Even though, to quote Ofqual once more, grades “are reliable to one grade either way”. And as I’ve said before, unreliable grades do great damage, and this is a problem that must be fixed.