This guest blog has been kindly written for HEPI by Gervas Huxley of Bristol University and Mike Peacey of the New College of the Humanities.
Parents of undergraduates frequently express surprise at how little time their children spend in lectures and classes. On open days it is common for both pupils and their parents to ask for information on contact hours. It was concerns of this kind that led us to use the Freedom of Information Act to collect data on both contact hours and class size. This allowed us to compare the teaching arrangements at UK universities in more detail than any study since the 1963 Robbins Report.
Despite the widely held belief that contact hours are low we found that since 1963 average contact hours are have not changed very much, while class size has increased very substantially. We also found large variation in how much teaching students receive both between and within subjects. In response to these findings we proposed that the government introduce an input-based metric that weights contact hours by teaching intensity making it possible to compare teaching delivered in different ways at different universities. The Government responded to our proposals with a pilot study to investigate how practical and useful the collection and publication of this information would be.
The pilot study confirms our finding that there is a large variation in how much teaching students receive. However, when the Office for Students published the key results of the pilot study they concluded that teaching intensity imposes a ‘significant burden’ and was ‘not found to add value’. This conclusion is supported by almost no explanation or evidence. In addition to the OFS, Vice Chancellors and Professors of higher education have been queuing up to repudiate the suggestion that students are entitled to information on how much teaching they can expect at university.
In their influential study Academically Adrift (2011), Arum and Roska discuss the barrier to any attempt to increase the accountability of Universities:
As knowledge producers academics are trained to use their pens effectively and would no doubt be quite persuasive in op-ed articles, policy reports and book monographs about the risks and undesirability of any externally imposed accountability schema.
In this blog we explain why we believe information on teaching intensity should be collected by the Higher Education Statistics Agency and be made available to students and other stakeholders. We begin with an example.
Suppose you are buying your first car, it’s the most expensive purchase you have ever made. When you ask the sales person for details about the brakes, you are informed in a patronising tone that you really don’t need to worry about the brakes because “the car is fitted with all the latest safety devices. It has a state-of-the-art air bag, side impact protection, and crumple zones.”You reply that this is all very well but surely the vehicle needs to be fitted with brakes? The sales person does not actually answer this question, instead tells you that what matters is the “driving experience”. You reply that you agree that the driving experience is important, but you still want to be given information about the brakes. By now the sales person is treating you as if you were about five years old. Look he says, “owning a car is a transformative experience – I have spent years reflecting on what it means to drive, my expertise is informed by 50 years of research and we know that one cannot judge the overall quality of the driving experience by the specification of the brakes”.
The arguments that have been used to criticise teaching intensity are to paraphrase the theoretical physicist Wolfgang Pauli ‘so bad they are not even wrong’. No sensible person is likely to disagree that what takes place inside the classroom matters: Should lectures be flipped? Should learning be collaborative? We can all agree that the choice of pedagogy is important. The perfectly valid claim that how we teach matters does not imply that how much we teach does not. Perhaps the most common objection to our proposal is that what matters is ‘quality not quantity’. This implies that quality is independent of quantity. With respect to class size this claim is especially dubious. Students in small classes are likely interact more with academics and if interaction of this kind is thought to be one of the defining characteristics of a high-quality education quality will depend on class size and therefore ‘quantity’.
Where we agree with the OFS and the TEF panels that what ultimately matters is learning gain. HEFCE define learning gain as“an attempt to measure the improvement in knowledge, skills, work-readiness and personal development made by students during their time spent in higher education.”These measures “help universities and colleges understand the effect of different teaching practices.”We have no objection in principle to the claim that it is outcomes, not inputs, that ultimately matter. If we had truly reliable measure of how much students learn we could dispense with any measure of inputs – regulation could focus exclusively on comparison of learning gain and cost. No input-based metric of any kind would be necessary. It is not just teaching intensity, the use of flipped classrooms, collaborative learning, qualifications for teachers – could all be disregarded because their contribution would be picked up by differences in the chosen outcome measures.
In addition to measuring learning gain we must also assess the cost effectiveness of different teaching practices. In their 2003 paper Funding Higher Education in the UK: The Role of Fees and Loans Greenaway and Haines made the case for the introduction of tuition fees. Two decades of underfunding had resulted in a £3 billion ‘funding gap’, and one consequence of the shortfall was that “average class sizes in universities have doubled in the last 20 years. This has led to ever larger lecture classes and the disappearance of small group tutorials in many universities.” These changes had the potential to “degrade the learning experience and threaten the quality of education.” The 1997 National Committee of Inquiry into Higher Education (usually referred to as the Dearing Review) that made the case for the introduction of tuition fees states that ‘The rise in student numbers over the last decade … has meant larger class sizes and less contact time for students. … research indicates that students perform worse in large classes.’ If it were true that teaching intensity has no bearing on learning gain, then it is hard to see how a compelling case for the introduction of tuition fees could have been made.
What does the evidence tell us? Unfortunately, as all sides in this debate should acknowledge we know less about learning gain in higher education than in schools because the data needed to evaluate competing claims is not collected. To measure learning gain we need to be able to compare outcomes across institutions. This requires standardised tests taken by students at different universities. In higher education, proposals to introduce standardised tests have been blocked by the sector and not much data on contact hours and class size have been collected. This means that we have very little robust evidence of learning gain in higher education.
In schools standardised test scores provide a measure of outcomes that allow researchers to measure the importance of different inputs. Perhaps the most important finding from this research is that in the context of a school’s teacher quality is more important than class size (Hanushek 1999). This result needs careful interpretation. It does not imply that class size is of no importance. For sufficiently large variation, the common-sense belief that class size matters is supported by robust evidence based on randomised controlled trials. These show that if class size is reduced by as much as 30% pupils will benefit (Schanzenbach 2014). However, in schools, because both the length of the school day and class size are difficult to change, studies usually find small variation in contact time and class size and meagre benefits from affordable class size reductions. In the context of US schools Paul Krugman in his New York Times blog argued that when governments wish to reduce education budgets it is not class size but teacher pay that gets squeezed.
As economists we need to understand what is happening on different margins, and that policy must focus on the right margin. In schools it is right to concentrate on teacher quality because the payoff to improving teacher quality is greater than the payoff to reducing class size. One reason for this is the prohibitively high cost of significant reductions in class size and the finding that variation in class size is small.
The TEF pilot study which collected data on the Gross Teaching Quotient (GTQ) confirms our finding that there is large variation in teaching intensity. It also conclusively demonstrates that only administrative data can provide reliable information on how much universities teach. Because the TEF panel are unable to link the GTQ data with any standardised test scores, it has not been possible to draw any conclusions from the pilot study about whether or not class size matters for learning gain.
What we do know is that in higher education variation in class size is much larger than in schools. We found that when considering the middle 50% of the distribution of mean class sizes, different subjects varied between 70% and 500%. We believe that if class size variation in schools was this large, nobody would argue that class size does not matter.
The enormous variation in teaching intensity that we find has echoes in Howard Bowen’s ‘revenue theory of cost’ (Bowen 1980). Bowen demonstrated that asking ‘how much does college cost?’ is like asking ‘how long is a piece of string?’. There is no meaningful answer: “at any given time, the unit cost of education is determined by the amount of revenues currently available for education relative to enrolment.”
The crucial difference is that in our data, there is almost no variation in revenue per student. This means governments set the level of funding per student at almost any level they choose, and for a given level of funding universities can provide as much or as little teaching as they chose. The degrees of freedom in the system are even larger than Bowen suggested.
It would certainly be good news for the Treasury if it could be shown that class size is unrelated to learning gain. At a time when UK universities are once more defending their budgets, Vice-Chancellors and Professors of higher education who promote this argument should be careful what they wish for.
For us the key objective is to increase accountability and transparency. Schools, hospitals and corporations are all subject to rigorous reporting obligations and we see no reason why universities should not provide comparable accountability to students, parents and taxpayers. The current system allows universities to report contact hours with no way for stakeholders to know if the contact hour comprises one lecturer in an auditorium with 200 students or one lecturer providing guidance and mentoring to three students in a seminar. Our metric provides a simple way of addressing this omission.
We do acknowledge that reporting teaching intensity will entail some additional costs on university administrators. But given the truth of the old adage that you can’t manage what you can’t measure, we are surprised that universities don’t perceive our metric as a useful aid to internal resource management. And, if internal governance is not sufficient reason to measure teaching intensity, then increasing accountability should be ample justification for an expense that is no more than a rounding error on the generous funding they receive from students, parents and tax payers.