This blog was kindly contributed by Paul Hayes, Deputy Vice-Chancellor, Dean Machin, Strategic Policy Adviser, and Paul Spendlove, Graduate Outcomes Manager, at the University of Portsmouth.
Earlier this year we wrote for HEPI about the unenviable challenge HESA faces in introducing its new Graduate Outcome Survey.
We were worried that HESA’s inexperience in interpreting the nuances of Standard Occupational Classification (or SOC) coding would lead to under-reporting the number of graduates in professional roles. We were also concerned that HESA’s decision not to use linked data-sets to establish which graduates were in further study would lead to some graduates in further study being counted as unemployed.
The Government’s announcement on its student number control shows the very real significance of accuracy here. Eligibility to bid for some of the extra student places requires, among other things, data showing that 75 per cent or more of a university’s graduates are in highly-skilled employment or further study. The 2020 Survey will not affect this year’s allocation but should the Government keep controls beyond the very short-term, 2020 Survey data could be used to determine directly how many students universities can recruit.
Were our concerns misplaced?
Predictions can be wrong and as we – like all universities – now have our data we can check. HESA’s approach with recently updated SOC data has achieved consistency across the sector so we got this one wrong and congratulate HESA. Graduates in further study, though, is a different story.
The new Survey was conducted fifteen months after graduation – or in the final weeks of study for anyone who started a twelve month postgraduate course the academic year following their graduation.
Their taught courses will have finished and most will be thinking ‘what next?’ They may no longer see themselves as students but as unemployed job-seekers. They will, though, be registered full-time students. What did these graduates say in the Survey?
University of Portsmouth data
Comparing HESA’s Survey data with our own records shows that 20 per cent of the graduates HESA records as unemployed during survey week were, at the time, registered with the University as full-time, postgraduate students. To put the point differently, the Survey over-reported our graduate unemployment by 25 per cent. This is not an insignificant inaccuracy.
And it could be worse. About half of our graduates who go on to further study do so at other universities. It is likely that some other graduates who HESA report as unemployed are also in further study. We do not have the data so we cannot know but, if our data is representative – if the inaccuracy is not Portsmouth-specific – HESA could have inflated the graduate unemployment figure by a further 10 per cent or 20 per cent.
Why does this matter?
Clearly, the inaccuracy of University of Portsmouth data on such an important value-for-money measure matters to us a great deal. But if our data is representative the wider impact is far more significant. HESA’s reputation will take a hit. More importantly, now – more than ever – we need accurate graduate unemployment data.
We have a Government that does not want the higher education sector to expand, and includes ministers as well key advisers who would prefer the sector to shrink. We might disagree with the wisdom of a smaller university sector but if changes are based on good evidence we can hardly complain. But if the evidence is not accurate and has the imprimatur of a respected and independent statistics agency, things are different.
What should HESA do?
Previously, we advised delaying the publication of the Survey until it is more accurate. HESA rejected this and we do not expect them to change their mind. HESA has promised to attach ‘health warnings’ to its statistical releases to advise data-users (that’s students, parents, policy-makers, journalists – in fact potentially everyone) on the appropriate and inappropriate uses of the data. HESA needs to make these health warnings clear, unambiguous and unavoidable. Otherwise they will be lost in the bright lights of the headlines and the league tables.
HESA must also urgently work on a revised set of Graduate Outcomes data for its statutory customers. There is still time to correct the significant misreporting of further study before the Government makes substantial policy decisions that could materially impact the sector and the sustainability of institutions.
HESA official response:
We realise that some providers have indicated concern about the reporting of further study.
Graduate Outcomes is a voluntary survey of graduates. Respondents are invited to answer a set of questions based on their perceptions and judgement using a questionnaire that has been extensively cognitively-tested. They make their own determination of their activities and what they consider to be the most important activity in the census week. That may include some graduates who have completed their active studies but are still registered as students during the census week, having commenced courses during the 15 months since gaining their original qualification.
To take this into consideration, data users will be able to optionally exclude graduates who are likely to have spent a majority of the 15 months interim period in full-time study within HESA’s published releases to focus on those who have spent the majority of the time in the labour market. The releases will explain why users may wish to do this. In this way, they can make informed choices about inclusion or exclusion based on their desired use of the data.
HESA plans to utilise linked student data when this is available, to provide a richer set of information about further study, but this is not currently available in the right timeframe for HESA’s main Graduate Outcomes publications.
I don’t think its helpful to suggest HESA’s (Oblong’s) SOC coding is now consistent when it isn’t, remains highly inaccurate in some cases and requires urgent attention to improve data quality for subsequent waves of the survey.
This is very useful, and adds to a body of information on how innappropriate and inadequate this measure is as a real record of students’ activity after a degree. Can you point me at similar resaerch on howmany students achieve “graduate level” employment more than a year after graduating after time spent in a “lower” level job?
Wow this is a shocker! I look forward to a HESA response.
Sadly the review of SOC data did not achieve consistency across all of the sector. Occupations identified as having systematic coding errors have not been corrected for some providers, and there are still inconsistencies in coding for the same job titles in the same record, therefore still leaving inconsistencies in Year 1 data. Add to this the issues raised with regards to further study data and the outcomes become even less representative. Unfortunately any further issues with SOC will have to wait until year 2 but hopefully HESA will address the further study issues for year 1 data.