This blog was kindly contributed by Dennis Sherwood, who has been tracking the goings on at Ofqual in relation to this year’s public exam results for HEPI. This is the ninth in the series as we rapidly approach the release of the GCSE, AS and A level grades in the next three or four weeks. You can find Dennis on Twitter @noookophile
The slide pack presenting the key messages from Ofqual’s 21st July Summer Symposium is an important document. Its purpose – presumably – is to reassure students, teachers, parents, carers, colleges, universities and employers about the robustness and fairness of this year’s exam-free GCSE, AS and A level grades, to be announced on 13th and 20th August.
Slide 12, reproduced here, is of particular importance: it is the evidence that the grades teachers have been asked to submit – Centre Assessment Grades (CAGs) – have resulted in a distribution significantly more ‘optimistic’ than recent history, so driving grade inflation.
That is important, for “no grade inflation” has been a cardinal feature of Ofqual’s policy since 2012. As a consequence, the exam boards, unilaterally and without consultation, will over-rule the schools’ submitted grades: in August, the grades that candidates receive will be those determined by the results of Ofqual’s statistical, and largely historically-based, model, the details of which remain obscure.
To limit grade inflation, it is therefore highly probable that many – perhaps as many as 30% – of the submitted grades (which number rather less than six million) will be down-graded. 30% of six million is about 1.8 million. That’s a far larger number than the number of candidates (remembering, of course, that the great majority of candidates sit more than one subject), so it’s very likely that every young person in the country will be awarded at least one grade lower than the grade their teachers submitted on their behalf. That could lead to trouble – big trouble – especially since the current rules for appeal are narrow and technical, and (unlike in Scotland) do not allow for appeals on the grounds of unfairness.
Given the significance of this slide, it should be clear, and contain complete and correct information.
With those thoughts in mind, this is my ‘examiner’s report’.
Point 1 – What do the columns mean?
It is good practice, as recommended in the Government’s own guidelines, to ensure all data items – such as the columns and rows in a table, and the axes of a graph – are clearly defined. What do the columns mean?
Anyone reading the chart might answer ‘the percentage of students awarded the corresponding grade’.
If that is the case, then 91.0% of 2019 A level candidates were awarded grade D. That is unlikely. So the columns must represent something else, for example – and I’m guessing, for the chart doesn’t tell me – the cumulative percentage of students awarded the grade identified at the top of the column, and all higher grades. That’s plausible, for 91.0% of A level candidates might well have been awarded one of grades A*, A, B, C or D.
It’s also highly misleading, for a quick glance might result in the reader thinking that, for A level, 13.4% of grade Bs were overbid. This is not the case. The percentage of candidates actually awarded grade B in 2019 was 26.1% of the total 2019 cohort; the corresponding number for this year’s CAGs is 27.2%. The difference is only 1.1%, not 13.4%. But I had to work that out for myself.
Point 2 – What do the numbers mean?
In presenting data, there is no place for the ‘obvious’. That the entries in the tables are percentages, rather than just numbers, is ‘obvious’ only to ‘the initiated’. The table should not be presented on the assumption that only ‘the initiated’ will look at it: the nature of the numbers should be explicitly stated so that everyone can immediately understand.
Point 3 – What is the third row?
The third row in each of the three tables is not defined. I need to work out for myself that it is the result of subtracting the number in the first row from the corresponding number in the second row in the A level and GCSE tables; but, confusingly, the second from the first in the AS table (of which more shortly).
Point 4 – What does the third row mean?
Looking across the third row in each of the three tables, there is a steady rise, followed by a steady fall. What is the significance of that pattern? What does the third row actually mean?
Point 5 – The AS table
The rows in the AS table are the other way around. That’s not only sloppy, but also, as noted in point 3, highly confusing. And why are the row heights different in each of the three tables? That’s pretty sloppy too.
Point 6 – The GCSE table
Why are there some blanks? Are the corresponding entries zero? This is very confusing too, especially since the CAGs row is complete.
Point 7 – The A level comparison is misleading
As far as I understand this year’s process, the statistical standardisation algorithm is likely to compare A level CAGs to the corresponding average over 2017, 2018 and 2019. So why is only 2019 shown as the comparison on Slide 12? Why not 2017 or 2018? Or – much better – the average?
Reference to authoritative sources of historic grade patterns, such as JCQ and Stubbs, will show that the distribution of grades in 2017 favours rather higher grades as compared to 2018, which in turn favours higher grades than 2019. Comparing this year’s CAGs against only 2019 portrays them in the worst possible light. Yet it is this comparison that was – presumably deliberately – chosen.
According to the (cohort weighted) average over the last three years, the percentage of candidates awarded A* to C is 76.7%, as compared to the figure of 75.8% shown on the slide – 0.9% closer to the CAG figure of 87.0%.
The CAGs are still over-bid, but that 0.9% is ‘interesting’ in that it is nearly one-half of the ‘about 2%’ hinted at for this year’s allowed grade inflation – and the other half is largely accounted for by the year-on-year variation of the A* to C % (75.8% in 2019, 77.0% in 2018 and 77.4% in 2017).
Point 8 – The AS comparison is misleading
By the same token, the AS CAGs should be compared not to 2019 alone, but to the average over 2017, 2018 and 2019.
Point 9 – The GCSE comparison is misleading
Likewise, the GCSE CAGs should be compared to the average over 2018 and 2019.
Point 10 – Where do the numbers come from?
When numbers are shown in a table or on a graph, it is good practice to identify the sources. But the sources are nowhere to be seen.
But those three, sparse, numbers for GCSE are a puzzle. Reference to JCQ or Stubbs will show that, for 2019 GCSE in England, 20.6% of candidates were awarded grades 9, 8 or 7; 67.0%, grades 9 to 4; 98.3%, grades 9 to 1.
But the corresponding numbers shown in the table are 24.7, 72.7 and 98.5 (I deliberately omit the % symbol to make a point).
I’ve searched high and low for that set of numbers – in individual 2019 subjects, and in 2018, but I just can’t find them.
Where do they come from? What is the significance of these numbers, and – more importantly – the difference between these and the corresponding CAGs? Assuming, of course, that those numbers are indeed the CAGs…
Slide 12 is a model of bad practice rather than good. And as for quality control…
Indeed, if this chart were presented in an examination, it would be given a very poor mark – an irony indeed given Ofqual’s role.
Some might consider my points – such as the absence of the % symbol, or the ordering of the rows in the AS table – as trivial, picking nits. Not so. There is an obligation on anyone presenting data to be complete and explicit, in all details. In an exam, the imperative of including the % symbol will be in the mark scheme. Not everyone reading the slides is a confident statistician. The information must be clear and explicit, and easily and correctly understood by anyone looking at the slide, not just the expert who knows how to read between the lines – or indeed numbers.
But two points in particular are fundamentally important.
Point 7, the failure to compare A level CAGs fairly against the three-year average, is, to me, a deliberate distortion, with intent to mislead. Ofqual knows that their statistical model uses an average, yet they present only a single comparative year – which just happens to be the one year that portrays CAGs in the worst possible light.
Point 10, the GCSE numbers, is even worse. Are those just arbitrary numbers, pulled out-of-the-blue? Everyone reading that chart (once they’ve worked out point 1, that the numbers are cumulative) will infer – and believe – that 24.7% of students for 2019 GCSE were awarded grades 9, 8 and 7, as indeed reported in The Times; the erroneous figure of 72.7% for the award of grades 9 to 4 is reported in The Times, TES and inews. Yet authoritative sources show that these numbers are 20.6% and 67.0% respectively. No one – other than an obsessive nerd like me – will ever dream of checking the numbers to original sources; everyone will take them on trust. But if just one number is wrong, that trust is broken. How many other numbers in the slide pack might be wrong?
It may be that all the points I have made were addressed in the Symposium. But I wasn’t there. I have seen only the slide pack I downloaded. Good practice is for the slide pack to be intelligible and complete in its own right.
And while talking about the slide pack, another puzzle.
I accessed the Ofqual website soon after the slides were posted, and downloaded a file with the filename 2020_Summer-symposium_slides_amended_210720. I went to the website just a minute ago, and downloaded a file with the filename 2020_Summer-symposium_slides_210720. The filenames are different, as are the files: the earlier (‘amended’) version has some slide notes that do not appear in the version now available. Why have the notes been deleted?