This blog on yesterday’s announcement about this summer’s public exams has been written by Dennis Sherwood, an independent consultant. He has previously produced much of HEPI’s past output on A-Levels, including 1 school exam grade in 4 is wrong. Does this matter?
I’ve just been reading about the proposed arrangements for awarding school exam grades, now that schools have been closed and exams cancelled. The key themes are:
- Teachers will be asked to submit their recommended assessments, which will then be ‘processed’ by the exam boards and Ofqual to determine the awards.
- School league tables will not be published.
To my mind, this is realistic and pragmatic – teachers know their students well and are in the best position to make fair and wise judgements, and the knowledge that those judgements will not impact their school’s league table position removes the corresponding pressure to game the system.
How this might work in practice? I don’t know, for few details have been published. But if I were asked to contribute to the design of the ‘process’, one possibility might be for teachers to be asked to submit, on behalf of each student, a recommended grade from U to 9 for GCSE and from U to A* for A level. The exam boards and Ofqual could then check the overall distribution of any school’s recommended grades against corresponding distributions from previous years, and how A level recommendations compare to GCSE and AS level actuals.
Spot checks might then be carried out on the school’s evidence for any outliers, and perhaps for general due diligence too. Personally, I would not be comfortable discriminating between ten GCSE categories, and A level is only marginally less problematic with seven. My preference would be for, say, four – but let me assume here that the ‘process’ requires teachers’ recommendations to reflect the ten GCSE and seven A level grades currently used.
And then I had an intriguing thought. Might teachers do a better job of identifying the right grades than the exam system?
Although I know of no evidence that teachers get this right, there is real, compelling, evidence that the exam system gets it wrong – evidence that comes from the most authoritative source possible, the exam regulator, Ofqual. Here is an extract from an announcement posted to the Ofqual website on 11 August 2019:
more than one grade could well be a legitimate reflection of a student’s performance
This statement was written in response to an article that had appeared in that day’s Sunday Times about the unreliability of A level grades. Ofqual’s statement is unqualified, and so, presumably, applies to all subjects. And its meaning is clear: any grade actually awarded to any student in any subject is just one of an (undeclared) number of grades, each of which is ‘a legitimate reflection of a candidate’s performance’.
In general, this is important but – in this particular instance – even more so. For if the exam system cannot reliably determine whether a particular student merits, say, a grade 5 or a grade 6 for GCSE Geography or an A or an A* for A level History, why might teachers do any better?
Let me answer my own question: because it is, I think, quite possible, if not quite likely, that a teacher is better able to do this than the exam system. After all, teachers have much more information – information accumulated over their careers in general and information about each individual student gathered over, say, two years in particular.
But there are three conditions that must hold.
- First, that teachers behave with honesty and integrity.
- Secondly, that teachers are trusted.
- And, thirdly, that teachers are not pressured into breaching that trust by the so-called ‘unintended consequences’ – the perverse incentives, of performance measures such as league tables.
Indeed, if teachers are trusted, if teachers earn and maintain that trust, if the performance measures encourage the ‘right’ behaviours rather than the ‘wrong’ ones, then the entire examination process could be transformed: exams would probably not be thrown away altogether, but the balance between a teacher’s fair assessment of years of work and what happens in a few sweaty hours in May would be tipped significantly towards the former.
I believe that this would be a good thing, but others will have different views (great – please post a comment). There is certainly a suspicion, and probably hard evidence, that self-assessment results in bias. But right now, there is an opportunity to carry out an experiment that no one would ever have thought possible: to base candidates’ grades primarily on teachers’ assessments for real.
So there’s the challenge. Teachers, step up to the plate, and prove to everyone that, yes, you can do this with honesty and integrity. Demonstrate, as individuals and a profession, that, yes, you can be trusted.
And if that can be seen to happen this year, that sets a precedent – to do this routinely in the future. Over the coming year, while exams are off the table, there is an opportunity to do some other important things too: for example, to determine ways in which an assessment of a single teacher can be sensibly moderated to address the important issue of bias; to consider, seriously, scrapping GCSEs altogether; and – my own bug-bear – to ensure that the grades for all school exams are fully reliable.
Fundamentally, though, it is vital that the teacher assessments to be used over the coming months are, and are seen to be, honest and trustworthy. Yes, it’s all a matter of trust.
It’s all very well in theory. And yes, for this year the best we can do.
But …
Teachers’ perceptions of attainment may well be significantly affected by things that are not about attainment as such – work ethic, “bright manner”, conformism sometimes – not to mention stereotypes.
We can avoid this by a more formulaic approach to assessing – eg basing on internal exams – but then what do we gain over external ones?
Parent pressure – we have suffered years and years of disempowering schools from standing up to parents. If we go to teacher assessment then there will be enormous pressure to up the grades of those with pushy parents – which will work against widening participation targets
And finally – the educational benefit of revising for exams. Yes, there is one. It makes you pull the material together and really learn it properly – which is surely what we want to happen. And no, it won’t happen by and large unless the exams are high stakes, unfortunately. That’s a casualty of the increasing tendency to instrumentalist education rather than to promote intrinsic motivation.
Hi Cath
Thank you; yes, everything you say is important and valid.
Let me just clarify that I am not advocating scrapping all exams. That said, I think that GCSE is a special case, and very much a historical left-over, so I believe its continuation needs to challenged; an exam on leaving school – say, A level – is a very different matter, and does play a valuable role.
But I am advocating asking whether, right now, teacher assessment might – perhaps surprisingly – be more reliable than GCSEs and A levels as currently delivered.
As I have discussed elsewhere (for example, https://www.hepi.ac.uk/2019/01/15/1-school-exam-grade-in-4-is-wrong-does-this-matter/), on average across all subjects and across GCSE, AS and A level, about 1 grade in every 4 is ‘wrong’ – meaning that about 1 grade in 4 would be changed if all the year’s scripts were to be fairly re-marked by an appropriate senior examiner.
And for subjects such as English and History, the reliability of grades 1 to 8 at GCSE, and A to E at A level, is about 50% close to the grade boundaries, rising to about 65% in the middle of the grade widths (see, for example, https://www.silverbulletmachine.com/single-post/2018/12/06/Visualising-grade-unreliability).
I wonder if teacher assessments, on average, are better than 1 grade in 4 being wrong? Personally, I would not be surprised if this were indeed the case, and it would be great to gather some evidence.
To me, the ideal is an assessment process in which teacher judgement plays a much stronger role, alongside appropriate – and reliably graded – exams. And yes, as you rightly say, that’s all very well in theory. The challenge is to make it real.
Which is why your points (and I’m sure many others too) are so important, and why I wrote my penultimate paragraph: to use this unexpected, and of course most unwelcome, opportunity to do some good – to think this all through very carefully, robustly and professionally, and to find pragmatic solutions to the many problems that need to be overcome, such as those you mention. Much has already been done in this field, but perhaps the time is now right to make some real progress – and perhaps the policy-makers might be in the mood to listen.
Is there anyone out there who might like to contribute?
Dear Dennis,
You make a very strong case here, and I agree fully with your critique of exam-based assessment. However, I worry about rhetoric like:
“So there’s the challenge. Teachers, step up to the plate, and prove to everyone that, yes, you can do this with honesty and integrity. Demonstrate, as individuals and a profession, that, yes, you can be trusted.”
This personalises the issue too much for my liking. The problem isn’t that teachers lack honesty and integrity; the problem is that we all carry biases around with us, consciously or unconsciously.
The issues that Cath raises above are enormous – how can teachers NOT be influenced by
pupils’ work ethic, their behaviour and conformism? Parental pressure would also be irresistible.
There is a danger that reducing the issue to one of ‘trust’ simplifies a range of substantive psychological, cultural and social complexities.
Dear Steve (if I may…)
Your concern about my rhetoric is fair, for I am indeed advocating a particular point of view! And you’re right that my blog doesn’t offer solutions to the many important, real and practical problems – such as those identified by yourself and Cath – of designing, implementing and validating a teacher-led process of assessment that is trusted by parents, students and all the other stakeholders.
I wonder if one of the biggest problems is the trust of the student in the teacher’s judgement – [Chris’s] fear that “the [physics] teacher has it in for me” or that “[Alex] is teacher’s pet and bound to get a good grade”. I’m sure this problem is real, but is it truly insurmountable? Is it not possible to devise a wise solution? And yes, parents can be – and are – pushy. But is this pressure really irresistible? Could these problems not become surmountable, and resistable, if, for example, students and parents were involved in the process in some way?
That’s why, in my reply to Cathy, I suggested that there is now a unique and very precious “…opportunity to think this all through very carefully… and find pragmatic solutions to the many problems…”. It may be that Ofqual’s forthcoming guidance will resolve all this. Perhaps; but perhaps not. And if not, then might the teaching profession – for example, ASCL, NAHT, HMC…, working with leading educationalists – take a collective lead in designing a process that really does work?
This will not, of course, happen by itself. Someone has to take the lead. Hence my “step up to the plate” rhetoric!
..oops – sorry – Cath not Cathy!!!
Thanks for the thoughtful response, Dennis.
I certainly agree that this is an important conversation, and one that now needs to take place openly.
Really interesting and great stuff here. Thank you. Here are my thoughts for what they are worth
I think I won’t have too much trouble in predicting what my students would have got in the exam but if the exams are given less importance, it begs the question of what am I predicting? Their overall attainment? As measured by what? What does that actually mean? Obviously I agree with everything you say and teacher assessment is much more reliable but if it were to take on a much more important role, there would come a time when we would be asking ourselves what is it that we are assessing? Not that that would be a bad thing but I think it would be a trickier task than the one we have this year which is “simply” saying “What would this particular student have got in the exam if it had taken place?”
A simplistic solution would be a combination of teacher assessment and examinations; 50:50?
From personal experience both professionally and from being a mum to 2 teenagers, I have experienced a lack of trust and confidence at each level,within the education system itself. As you go from Pre schools to Primary schools and primary schools to secondary ,neither setting seems to have the trust in one another, and truly rely upon eachothers professionalism e.g pre-school professionals spend around 3-4 years studying for their early years qualifications and have valuable information to share with the Primary schools. Sadly,when assessments of key children are sent to Primary schools they are often viewed by some, as biased opinions. 4 weeks prior to year 6 SAT’S, my sons Primary school invited those less likely to achieve the government expectation, to a Breakfast club. These students had previous SAT’S papers drummed into them and then went on to get somewhere nearer to the expected grade.Sadly when these students reached senior school they were set by these results. They went in feeling like the high flyers only to be cut down after a few months as they realistically couldnt cope with the workload that was set. We even had a parents evening in year 9 where our sons English teacher was happy to inform us that the secondary schools were aware of underhand tactics that go on within primary schools to get these results. So with this in mind, how can we totally rely on Teacher assessments deciding the grade structure for the students at GCSE level, when this could potentially set kids up for a massive fall when they reach A level. A 50/50 approach I believe would be something to consider, however, Exams are primarily an important factor in a students life, as they need to appreciate that its a tough world out there and they have to work hard through challenging times if they really want and deserve to be there. Encourage kids to dream big and strive for what they want but also to be realistic of their strengths and what they are realistically capable of achieving. Otherwise sadly disappointment could be directly around the corner for the kids.
Hi Samantha – thank you.
I’m sure you’re right in that exams will not be discarded. I wonder, though, if there is now an opportunity to do some fundamental re-thinking, about the curriculum, about assessment, about the balance between teacher judgement and exams, and about wider issues such as the ‘joined-up-ness’ of the educational system as-a-whole, and relationships with other systems such as housing, child care and the wider society and economy.
Your description of the world-as-it-is-now is undoubtedly realistic. But must this ‘world’ be perpetuated for the next few decades? Or might it be possible to change it for the better?