As we await the outcomes of Dame Shirley Pearce’s review of the Teaching Excellence Framework (TEF), it is worth reflecting on what Sir Philip Augar’s Post 18 Review of Education and Funding tells us about the possible future of the TEF.
In doing so, it is first worth considering four challenges faced by the TEF that have informed the Pearce review.
- The current four purposes of the TEF (informing the choice of university applicants; raising the esteem of teaching; recognising and rewarding excellent teaching; and ensuring that higher education better meets the needs of employers) appear to be in tension with each other.
- The costs of running the TEF are seen to be too high. For example the UUK estimate that if the subject TEF was rolled out across the sector then it would cost around £40 million compared to £25 million for the institutional TEF.
- Despite these costs, these versions of the TEF provide information at the institutional or subject level rather than the particular degree programmes that applicants are choosing between. This is particularly problematic because the quality of teaching is determined at the programme level rather than at subject or institutional level.
- There have been sustained criticisms of the TEF metrics, particularly the dominance of employment measures given that there is no evidence that employment outcomes are related to the quality of teaching. Finally, the indications from the DfE’s TEF evaluation is that applicants mistakenly believe that the TEF involves inspections of teaching, which suggests they might not trust the TEF if they better understood the basis on which the awards are given.
When considered in the light of these challenges, it becomes clear that the Augar review does not take a consistent position on the TEF. There are two aspects to this. First, the review explicitly welcomes the TEF and, in its definition of low value degree courses in terms of those that offer low graduate premiums, its underlying logic is clearly aligned to the TEF’s focus on employment outcomes. However, the review also strongly signals the problems with the TEF metrics and particularly highlights the Royal Statistical Society’s criticisms of the metrics. This suggests confusion on the part of the review because these criticisms include the lack of any evidence for a link between graduate outcomes and teaching quality.
Second, the Augar Review’s emphasis on a holistic tertiary system could be read as suggesting an alternative approach to the TEF that covers the whole of the tertiary system and helps applicants to choose between traditional further education and higher education provision. The Augar review writes very approvingly of the kitemarking of level 4 and 5 courses and suggests kitemarked course should be allowed to charge the same level of fees as degree courses. This raises questions about whether this might be an alternative approach to the TEF, which might be more look like an old style QAA approach to measuring teaching quality.
So what does the Augar Review tell us about the future of the TEF? First, it suggests that there is very likely to be something that fills a TEF-like space. However, this could look very different according to whether its focus is on the graduate premiums/value for money side of the Augar review or the teaching quality side. The problem with focusing on the graduate premiums side is not just that it tells us nothing about teaching quality. It is also because of the time lag between the data being provided to applicants and when they will actually enter employment after graduation.
There are two aspects to this. First, applicants are being given data about the salaries of graduates who studied at an institution at least five years before the applicant will be studying there. High quality degree programmes are very likely to have changed significantly in this period and any institution or programme that hasn’t is likely to be of questionable quality. Second, given that the uncertainty around the economy, this time lag could mean that the information about graduate salaries is highly misleading. What will happen if applicants select courses purely on the basis of their graduate premiums and these premiums have disappeared once they graduate? Given it is a government requirement for this information to be provided to guide applicants’ choices, who would be responsible for ‘misselling’ these degrees?
These problems suggest that the Pearce review would be better to recommend a TEF that is focused on examining teaching quality and that recognises the limitations of any metrics that are used to inform its judgements. Such a clear focus would provide the space in which to develop an approach that provides applicants with more trustworthy information about the quality of the degrees they are thinking of studying.
Email: [email protected] Twitter: @paulashwin
The Teaching Excellence Framework (TEF) needs a name change to explain what it actually does and what the “scores” mean.
The review should revisit what the “purpose” is of the TEF and start with the end in mind.
Like so much of Government activity, the desire to measure outcomes and return on investment is often an after thought arising from a lack of trust and a muddled birth.
The real issue is sometimes a failure by funders to clearly define the objectives given to the institutions they want to monitor and evaluate.
Universities are a case in point.
As there seems to be no wide agreement on what “providing a good education” looks like, it’s not surprising that what can be easily measured might not equate to what is appropriate.