The perfect storm: AI, assessment and a sector under pressure

Author:
Ciaran Donaghy and Rebecca Robinson
Published:

This blog was kindly authored by Ciaran Donaghy, Lead Policy Officer (Devolved Nations) and Rebecca Robinson, Data Analyst (Higher Education Insight), both of QAA.

Everyone’s talking about it – and has been doing so with increasing urgency over the last three years. From scandalised press commentators, anxious moral philosophers and concerned policymakers, through to lecturers, students, pedagogic experts, quality professionals and learning technologists, the impact of generative artificial intelligence on higher education has been one of the defining conversations of the sector in recent years.

It may therefore come as a surprise to those outside the sector to discover that there is no single response to the proliferation of GenAI tools in higher education. There are, of course, a range of institutional responses – and that variability is something to be expected in a sector that celebrates the diversity and autonomy of its institutions. But what our new State of the Nation report, The Perfect Storm: AI, assessment and a sector under pressure, finds is that the variability runs deeper than institutional differences. It runs through the practices of different academic departments, programmes, modules and individual practitioners within institutions, resulting in confusion among staff and students, and real inconsistencies in the learner experience.

GenAI continues to stymie the sector – and collectively, we’ve shied away from a comprehensive sector response, perhaps because it feels too big, too impossible to come up with an approach that accommodates all the various nuances and diversity higher education contains. Some variability is good and necessary. Adapting to subject content, assessment methods and student characteristics can serve learners well, and a sector in rapid transition in response to the challenges posed by GenAI is always going to be one in which practice is uneven. But where variability is driven, not by pedagogy, but by mismatched policies; lack of clarity for students; gaps in staff confidence and understanding; or differences in student access to and skills with AI tools, it becomes a fundamental risk to the conditions under which academic standards can be assured.

Our report examines and collates the findings of a range of recent studies of GenAI use in higher education and the issues and trends identified by QAA’s own review reports, alongside perspectives gleaned from a series of roundtables we held with student and sector staff stakeholders this spring.  

The report identifies several key risks. There are real questions around the validity of assessment design when AI use can’t be reliably verified. There are concerns about foundational skills and about the meaning of a degree when some graduates may be achieving qualifications without engaging substantively with their studies. But perhaps the most striking finding is the gap between institutional policies and their implementation on the ground.

This gap matters because it falls hardest on students. Students experience the unevenness of policy implementation before they experience the policy itself, encountering different rules and expectations within a single programme of study, with uncertainties about what is permitted and anxieties about potential breaches of institutional policy. These inconsistencies and uncertainties may most significantly impact upon the experiences and outcomes of international students, students for whom English is a second language, neurodivergent students and students lacking cultural capital and technological confidence in the legitimate academic uses of this technology.

Our report therefore urges institutions to address consistency of student experience as a priority. This issue represented the single most recurring concern identified across our review evidence and our direct engagement with staff and students. Providers may therefore wish to consider how their policies are operationalised in practice, and how legitimate variation between disciplines and modules is communicated to students in ways that don’t produce confusion.

We are very aware of the resourcing constraints currently affecting our sector and that many institutional responses to the challenges posed by GenAI can, by virtue of their emphatically human focus, prove to be particularly resource-intensive. We are nevertheless recommending institutions ensure staff and students have the skills and confidence to use these tools and for students and staff to engage in any such training opportunities. Training emerged as the single strongest need flagged by both our staff and student roundtable groups – training focused not only on how to use GenAI, but on how to use it well, affording due consideration of the ethical, environmental and cognitive concerns it may raise.  

We are also recommending that institutions should, from the outset, build student voice into the development of GenAI policy. Co-creation with students has repeatedly been identified as a feature of good practice in QAA reviews and was strongly endorsed by both our staff and student participants in our research.


Interested in AI? Here is some of HEPI’s publications on the topic:

Get our updates via email

Enter your email address to subscribe to this blog and receive notifications of new posts by email.

Comments

Add comment

Your comment may be revised by the site if needed.